skip to content

In Debezium, what does snapshot.select.statement.overrides change about a table's snapshot?

level: middleimportance: nice to knowfreq 28%

answer

  1. swaps the query used to read the table
  2. one list property plus one per table
  3. filters rows or trims columns
  4. the streaming phase ignores your predicate
  5. updates can arrive for unseen rows

basics

~20 s

It replaces the default SELECT that reads a table during the snapshot with a statement you supply, so you can filter rows or restrict columns. It affects the snapshot only; streaming afterwards still captures every change to that table.

solid answer

~40 s

By default a Debezium snapshot reads each captured table with a plain full-table SELECT. `snapshot.select.statement.overrides` takes a comma-separated list of fully-qualified table names, and for each one you add a companion property — `snapshot.select.statement.overrides.<schema>.<table>` — holding the SELECT to run instead. Typical uses: skip archived history with a `WHERE` clause, snapshot only recent partitions of a table with a decade of data, or avoid dragging a huge blob column through the copy. The critical caveat is scope: **this filters the snapshot, not the capture**. Streaming still emits every change to every row of that table, including rows the snapshot deliberately skipped. A sink that upserts by key handles that fine; a sink that assumes an update always follows a prior read event will break the first time an excluded row is touched.

code

properties · 4 lines
properties
snapshot.select.statement.overrides=public.orders
snapshot.select.statement.overrides.public.orders=SELECT id, customer_id, status, total, created_at FROM public.orders WHERE created_at >= '2025-01-01' ORDER BY id
# coarser alternative when a whole table can be skipped:
# snapshot.include.collection.list=public.customers

go deeper

for a junior

Recall that the setting replaces the query used to read a table during the snapshot, and that it is configured as a list of tables plus one statement property per table.

for a middle

Explain the scope boundary clearly: the predicate applies to the snapshot query only, streaming captures every change regardless, and the practical result is update events for rows the destination never saw created.

for a senior

Show that you evaluate it against the alternatives — a table-level snapshot include list, or a stream-now start with filtered incremental snapshots — and that you check the sink's behaviour for unseen keys before shipping the filter.

for a principal

Own the question of whether the destination should hold that data at all. A snapshot filter is a weak boundary because streaming ignores it; a real data-scope decision belongs in the capture set or in a governed transformation layer.

## What it is During the snapshot phase, Debezium reads each captured table with a straightforward SELECT over the whole table. `snapshot.select.statement.overrides` lets you substitute your own query for specific tables. The configuration comes in two parts: the property itself lists the fully-qualified tables you want to override, and one further property per table — suffixed with that table's qualified name — carries the SELECT statement. Qualification follows the connector's convention, so the suffix is schema-and-table on Postgres and database-and-table on MySQL. ## Why you would use it Three situations account for nearly all real use. **Volume**: a table holds ten years of history but the destination only needs the last two, and copying the rest would add hours to a snapshot during which the source retains log segments. **Cost**: a table carries a large payload column that the destination does not consume, so selecting the needed columns keeps the copy small. **Hygiene**: rows flagged as soft-deleted or belonging to decommissioned tenants should never reach the destination in the first place. There is a natural companion setting: `snapshot.include.collection.list` restricts which tables are snapshotted at all. Use that when the unit of exclusion is a whole table, and the statement overrides when the unit is rows or columns within one. ## The consequence people miss The override applies to the snapshot query and nothing else. The streaming phase reads the log and emits every change to every captured row, with no awareness of the predicate you wrote. So a row excluded from the snapshot is invisible at the destination until someone updates it — at which point an `u` event arrives for a key the destination has never seen. Whether that is harmless or damaging depends entirely on the sink. A sink that applies events as key-addressed upserts simply inserts the row and carries on; the destination gains a row that arguably should not be there, but nothing breaks. A sink that translates change events into literal UPDATE statements, or that maintains counts and expects every key to have been introduced by a prior read event, produces errors or wrong aggregates. Decide which you have before you write the predicate. The mirror-image mistake is the assumption that filtering the snapshot also filters the stream — that a `WHERE tenant_id = 7` override yields a single-tenant feed. It does not. To restrict what is captured at all, you need capture-set configuration (table and column include/exclude lists) or transformation and filtering downstream. ## Writing the statement safely A few practical rules. Return the columns the connector expects: if the captured column set is not restricted elsewhere, omitting a column from the SELECT can produce events whose schema does not match what streaming will later emit for the same table, and inconsistent event shapes are painful downstream. Make the predicate stable — a filter on `created_at >= now() - interval '1 year'` evaluates at snapshot time and is fine, but reasoning about which rows it admitted becomes hard later; a literal boundary date documents itself. Include an `ORDER BY` on the key when the connector's snapshot behaviour benefits from deterministic ordering. And keep the statement cheap enough to run against production — an override with an unindexed predicate can be slower than the full-table read it replaced, which defeats the purpose entirely. ## Where it fits among the alternatives If the goal is a shorter snapshot, weigh the override against simply not snapshotting the data at all: start the connector in a mode that skips rows and backfill with ad-hoc incremental snapshots, whose signal payload also accepts a filter condition. That path gives per-table control at request time rather than at deploy time, is resumable, and can be paused — advantages the static override does not have. The override remains the better tool when the filter is a permanent property of the pipeline (this destination never wants archived rows) rather than a one-off backfill decision. ## Interview framing This is a differentiator question rather than a screening one. The answer that lands is not the syntax; it is the boundary. Say what the override changes, say plainly that streaming ignores it, and name the concrete downstream consequence — an update for a row the destination never received an insert for — along with which kind of sink survives that and which does not.

  • You exclude rows older than a year from the snapshot. What does the destination see when one of those old rows is updated?
    An update event for a key it never received a read event for, because streaming captures all changes to the table regardless of the snapshot predicate. Key-addressed upsert sinks absorb this by inserting the row. Sinks that emit literal UPDATE statements, or that maintain counts assuming every key was introduced by a prior read, break or drift.
  • When would you use snapshot.include.collection.list instead?
    When the unit you want to exclude is a whole table rather than rows or columns inside one. It restricts which captured tables are snapshotted at all, which is simpler to reason about and cheaper to review. Reach for statement overrides only when the exclusion is finer-grained than a table.
  • Is a static override the best way to shorten a first snapshot?
    Often not. Starting in a mode that emits no rows and then issuing ad-hoc incremental snapshots gives per-table control at request time, resumes after a restart, accepts its own filter condition, and can be paused under load. Keep the static override for filters that are a permanent property of the pipeline rather than a one-off load decision.

saying these in an interview costs you the question

  • Thinks the override also filters the streaming phase
  • Expects a filtered snapshot to yield a single-tenant feed
  • Omits captured columns and gets mismatched event schemas
  • Writes a predicate with no supporting index on a huge table
  • Never considers what the sink does with an unseen key

context