Schema Evolution and the Outbox Event Router

## The envelope you start with Every Debezium change event carries a nested value: `before` (the row as it was), `after` (the row as it now is), `op` (`c` create, `u` update, `d` delete, `r` read-during-snapshot), a `source` block with connector, database, table, timestamp and log position, and `ts_ms`. That envelope is exactly what you want if you are building a stream processor that needs to see the transition. It is exactly what you do *not* want if the consumer is a sink connector that expects one field per column, or a downstream table that mirrors the source. ## What the SMT does `io.debezium.transforms.ExtractNewRecordState` is a single message transform that replaces the record value with the contents of `after`. A create or update event becomes a flat struct whose fields are the table's columns; the schema handed to the converter is the row schema rather than the envelope schema. Snapshot reads (`op: r`) flatten the same way, so a sink cannot tell a snapshot row from a streamed insert unless you deliberately keep the `op` field. Because flattening throws the envelope away, anything you still need has to be re-attached. `add.fields` takes a comma-separated list — `op`, `table`, `lsn`, `source.ts_ms` — and adds each as a top-level field prefixed with `__` by default. `add.headers` does the same thing into Kafka message headers instead, which keeps the value clean when the value schema is under contract. ## Deletes, tombstones, and the classic bug A delete produces two records: the change event with `before` populated and `after` null, followed by a genuine tombstone — same key, null value — so that log compaction can eventually reclaim the key. The SMT cannot unwrap a null `after`, so it has to be told what to do. With the default settings it drops both the delete event and the tombstone. The result is the single most common report from teams new to this transform: *rows deleted in the source stay forever in the sink table*. Nothing errors; the events simply never arrive. The alternatives are (a) let the delete through as a record with a null value, which a delete-aware sink such as the JDBC sink with deletes enabled will translate into a `DELETE` statement, or (b) rewrite the delete into a normal-looking row built from the `before` image plus a `__deleted` flag, which is what you want when the target is an append-only or soft-delete model and you would rather mark than remove. In Debezium 2.x these are governed by `drop.tombstones` and `delete.handling.mode` (`drop` | `rewrite` | `none`); later 2.x releases consolidated them into a single delete/tombstone handling property and deprecated the older pair, so check the property names for the version you actually run. ## Where the transform runs, and what that implies The SMT executes inside the connector's task, per record, before the converter serialises it. That has two consequences worth stating in an interview. First, it changes the *schema* the converter registers — after flattening, the subject for that topic holds the row schema, not the envelope schema, so turning the SMT on or off mid-life is a breaking change for every consumer of that topic. Second, per-record transforms cannot join, aggregate or reorder; if you need the `before` image to compute a diff, flattening has already destroyed the information by the time a downstream job sees it. ## When not to flatten Keep the envelope when consumers need the transition rather than the state: audit trails, change-log tables, CDC-driven SCD Type 2 loads that compare old and new values, or any consumer that must distinguish an update from an insert on business grounds. Keep it too when different consumers of the same topic disagree — flattening is a topic-wide decision, and the safer pattern is to publish the envelope and let each consumer project it, or to move to an outbox table so the published shape is a designed event rather than a row image at all. ## Operational checklist Decide the delete behaviour explicitly rather than inheriting the default. Re-add `op` and a source timestamp, because a sink that cannot tell a snapshot row from a live insert cannot order or debug anything. Verify the flattened schema against your registry's compatibility rule before deploying, since the change of shape is not backwards compatible with the envelope. And test a delete end to end in staging: it is the path that silently does nothing.