Why does a Debezium MySQL connector keep its own schema history topic, and what breaks if that topic is lost?

## Two topics that sound alike and are not Candidates conflate them constantly, so separate them first. The **schema history topic** is internal. Its name comes from `schema.history.internal.kafka.topic`, it is written and read only by one connector, and it exists so the connector can reconstruct table structures. Nobody else should consume it, and its contents are an implementation detail. The **schema change topic** is public. It is named after `topic.prefix`, it is controlled by `include.schema.changes`, and it publishes DDL change events so downstream systems can react to a source schema change — auto-evolving a target table, alerting a data team, gating a deployment. Deleting it costs you notifications; deleting the internal one costs you the connector. ## Why MySQL needs history at all A MySQL binlog row event contains a table map and packed values. It does not contain column names, and its type information is minimal. Reading the current `information_schema` is not sufficient either, because the connector may be replaying binlog entries written days ago, before three `ALTER TABLE` statements landed. Decoding a row written under the old structure with today's structure produces garbage or an error. So the connector maintains an in-memory model of every captured table, applies each DDL statement it encounters in the binlog to that model, and appends the statement plus the binlog position to the history topic. On startup it replays the topic up to its stored offset and is then able to decode any row event from that point forward. The same design appears in the SQL Server, Oracle and Db2 connectors for the same reason. Postgres is different. Logical decoding hands the connector relation metadata alongside the changes, and the connector refreshes its schema from the catalog when a relation changes, so there is no history topic in the Postgres connector's configuration at all. Being able to explain *why* the property is absent there is a good senior signal. ## The topic's operational requirements One partition, so the DDL stream has a total order. No compaction and no retention-based deletion, because history is only meaningful read from the beginning — a compacted history is a corrupted history, and this is the single most common way teams destroy it, by pointing the connector at a cluster whose default topic policy compacts or expires everything. Exactly one connector per history topic; two connectors sharing one interleave two unrelated DDL streams. And it must be backed up or reproducible as part of your disaster-recovery plan, because it is genuinely connector state, not derived data. `schema.history.internal.store.only.captured.tables.ddl` limits what gets stored to the tables you actually capture. It keeps the topic small and startup fast, at the cost that if you later widen the include list to a table whose DDL was never stored, the connector has no history for it and needs a fresh snapshot. ## What failure looks like The connector task fails on start, typically with a message that it encountered a change event for a table whose schema is not known to this connector, or that the schema history topic is missing or empty. Streaming does not resume; lag grows for as long as the outage lasts, and on MySQL the retained binlog is finite — so a long outage can end with the required binlog position purged from the server, which turns a recoverable incident into a mandatory re-snapshot. ## Recovery paths If the source schema has not changed since the connector's stored offset, you can rebuild history from the database's current DDL using the connector's schema-history recovery snapshot mode — named `schema_only_recovery` in earlier 2.x and renamed in later releases, so confirm the value for your version. The connector reads today's structure, writes it as the history baseline, and resumes streaming from the stored offset. The safety condition is real and load-bearing: if any DDL landed between the offset and now, the rebuilt history is wrong for the rows in between, and you will decode them against the wrong structure. If the schema did change, or the offsets are also gone, the honest answer is a fresh snapshot: drop the connector's offsets and history, re-snapshot, and accept the duplicate rows downstream — which is why sinks in a CDC pipeline should be idempotent, keyed upserts rather than blind inserts. ## What a senior candidate adds Monitor it: alert on connector task failure, on the history topic's configuration drifting to compacted, and on binlog retention versus current lag, because the last of these is what decides whether an outage is a restart or a re-snapshot. And keep the history topic out of any cluster-wide policy automation that rewrites retention on unlabelled topics.