skip to content

Once a table carries a deleted_at column, every read in the system has to exclude those rows. What mechanisms keep that filter from being forgotten, and where do the leaks typically show up?

level: seniorimportance: should knowfreq 45%

answer

  1. rename table + filtered view = default-safe
  2. ORM scope covers only ORM traffic
  3. RLS covers ad-hoc SQL and BI tools
  4. leaks: aggregates, joined parent, exports, caches
  5. partial indexes keep the plan honest

basics

~20 s

Do not rely on discipline. Make the default path filtered: rename the physical table and expose a filtered view, use the ORM's global scope, or use row-level security. Leaks show up in aggregates, joins to parent tables, batch jobs, exports, caches and search indexes.

solid answer

~60 s

Relying on every developer to remember `AND deleted_at IS NULL` fails at scale, so make the filtered form the default. **Mechanisms** - **Views**: rename the base table (`orders_all`) and create `orders` as `SELECT * FROM orders_all WHERE deleted_at IS NULL`. All existing code keeps working and is filtered by construction; deliberate access to dead rows must name the base table. - **ORM global scopes** (Hibernate `@SQLRestriction`, Rails `default_scope`): convenient, but they only cover queries that go through the ORM. - **Row-level security**: the engine appends the predicate to every statement, covering ad-hoc SQL and reporting tools too — the strongest guarantee, at some planning cost. - **Structural**: keep dead rows in a separate archive table, so there is nothing to filter. **Where it leaks**: `COUNT`/`SUM` in dashboards; joins where you filter the child but not the parent; outer joins where the predicate lands in `WHERE` and silently drops rows; batch jobs and CSV exports; materialized views; caches and search indexes that were populated before the delete; and uniqueness or lookup-by-natural-key paths. Also index for it: make hot indexes partial or lead with the flag, or the planner's estimates drift as dead rows accumulate.

code

sql · 10 lines
sql
ALTER TABLE orders RENAME TO orders_all;

CREATE VIEW orders AS
SELECT * FROM orders_all
WHERE deleted_at IS NULL;

-- hot index sized to the live set only
CREATE INDEX orders_customer_live_ix
  ON orders_all (customer_id)
  WHERE deleted_at IS NULL;

go deeper

for a junior

Know that the predicate must be on every read and that a view or framework scope can apply it by default.

for a middle

Name at least two enforcement mechanisms and the classic leak sites — aggregates, joined parents, exports.

for a senior

Compare view / ORM scope / row-level security by coverage and cost, cover derived data invalidation, and address the index and planner impact of accumulating dead rows.

for a principal

Treat it as an invariant that needs enforcement plus verification: default-safe access paths, an explicit escape hatch, automated tests and reconciliation — and a willingness to switch to archive tables when the burden outgrows the benefit.

## The problem Soft delete converts a structural guarantee (the row is gone) into a convention (everyone remembers a predicate). Conventions decay: new joiners, new services, ad-hoc SQL from analysts, a quick fix at 2am. Each omission is a silent correctness bug — no error, just wrong data on a screen or in a number someone reports upward. So the engineering question is not "how do I remember the filter" but "how do I make the unfiltered form the unusual one". ## Mechanism 1: filtered views (rename-and-view) Rename the physical table to `orders_all` and create: ``` CREATE VIEW orders AS SELECT * FROM orders_all WHERE deleted_at IS NULL; ``` Every existing query that says `FROM orders` is now filtered without being touched. Reaching dead rows requires deliberately typing `orders_all`, which is greppable and reviewable. Simple views are usually updatable, so writes still work; if not, add `INSTEAD OF` triggers or write through the base table explicitly. Downsides: two names to keep in your head, migration tooling that introspects tables can get confused, and some ORMs need help mapping a view for writes. ## Mechanism 2: ORM-level default scopes Hibernate's `@SQLRestriction` (formerly `@Where`), Rails' `default_scope`, Django managers and similar features append the predicate automatically. This is the lowest-friction option and catches the majority of application reads. Its limits are real: raw SQL, native queries, reporting tools, other services sharing the database, and data-migration scripts all bypass it. It also becomes awkward the moment you legitimately need the deleted rows (admin restore screens, purge jobs), and `default_scope` in particular has a habit of leaking into inserts and associations in surprising ways. ## Mechanism 3: row-level security Engines with RLS let you attach a policy to the table so the predicate is applied by the database itself, whatever client issues the query. That covers psql sessions, BI tools and every service. You then grant a `bypassrls`-style role, or a policy keyed off a session setting, to the few jobs that must see dead rows. This is the strongest guarantee short of not storing the rows at all; the cost is an extra predicate the planner must handle on every access and a policy surface that has to be tested like any other security control. ## Mechanism 4: do not mix states The structural fix: hard delete from the live table and insert into an archive table in the same transaction. Nothing to filter, constraints keep their meaning, the hot table stays small, and restore is an explicit, auditable operation. The trade is that restore is more work and cross-state queries need a `UNION`. ## Where leaks actually show up - **Aggregates.** `COUNT(*)`, `SUM(amount)`, dashboard tiles and monthly reports are written by different people, often outside the application, and are exactly where a missing predicate becomes a visible wrong number. - **The other side of a join.** Developers filter the table they are thinking about. `JOIN customers c ON …` without `AND c.deleted_at IS NULL` resurrects deleted customers through their live orders. - **Outer joins.** Putting a parent-side predicate in `WHERE` rather than in the join condition turns a `LEFT JOIN` into an inner join and drops rows that should have appeared with NULLs — a filter that is present but placed wrong. - **Derived and cached data.** Materialized views, denormalized counters, Redis caches, Elasticsearch documents and analytics extracts were all populated before the delete. Soft delete does not notify them; you need an explicit invalidation path, exactly as with a hard delete. - **Batch and integration paths.** Nightly exports, partner feeds, billing runs and data migrations are the classic leak sites because they are written once and rarely reviewed. - **Lookup-by-identifier paths.** "Find the user by email" must decide whether a deleted user counts — for login it must not, for signup collision handling it might. ## Do not forget the plan A predicate on every query changes the physics of your indexes. If dead rows accumulate, an index on `(customer_id)` now contains mostly rows the query will throw away, so more pages are read for the same result and cardinality estimates drift. Two fixes: make the hot indexes partial (`WHERE deleted_at IS NULL`), which shrinks them to the live set and lets the engine skip the filter entirely, or lead composite indexes with the flag. Partial indexes are the better answer where the engine supports them, and they pair naturally with the partial *unique* index the same schema already needs. ## Verification Whatever mechanism you pick, add cheap tests: a repository/integration test per entity asserting a soft-deleted row is invisible, a lint or grep rule flagging `FROM orders_all` outside approved files, and a periodic reconciliation query comparing a dashboard number against a known-correct filtered query. Correctness that depends on memory should always be backed by something automated.

  • What is the drawback of relying on an ORM global scope such as Hibernate's @SQLRestriction?
    It only applies to queries that go through the ORM's entity mapping. Native SQL, reporting tools, other services on the same database and migration scripts bypass it entirely, so the guarantee is partial. It also has to be escaped deliberately for admin and purge paths, which in some frameworks is awkward and easy to get wrong.
  • How do dead rows affect query plans, and what do you do about it?
    Indexes and table pages fill with rows every query discards, so the same result touches more pages and the optimizer's row estimates for the filtered predicate drift as the dead fraction grows. The standard remedy is partial indexes restricted to `deleted_at IS NULL`, which shrink the index to the live set and let the engine satisfy the predicate from the index definition; failing that, lead composite indexes with the flag and purge on a schedule.

saying these in an interview costs you the question

  • Answering "we just remember to add the filter" with no enforcement mechanism
  • Believing an ORM default scope covers reports, migrations and other services
  • Filtering the queried table but not the joined parent table
  • Ignoring caches, search indexes and materialized views, which never learn about the flag
  • Assuming query performance is unaffected as the dead-row fraction grows

context