How does Schema Registry store its data durably, and what is the role of the _schemas topic?
answer
- _schemas internal topic = the store
- single partition → total order → monotonic IDs
- log-compacted, tombstones for deletes
- leader is sole writer; followers forward
- replay from offset 0 into in-memory KafkaStore
basics
~20 sSchema Registry stores all schemas in a special Kafka topic called _schemas. It is a single-partition, log-compacted topic that acts as a commit log; each registry node reads it into an in-memory cache, so the registry itself holds no separate database.
solid answer
~50 sSchema Registry is **Kafka-backed**: it writes every schema, subject, version, config, and delete tombstone as a record to an internal topic named **`_schemas`**. That topic is **single-partition** (to give a strict total order needed for assigning monotonic global IDs) and **log-compacted** (so the latest state per key is retained indefinitely while obsolete updates are eventually removed). On startup each registry instance consumes `_schemas` from the beginning to rebuild its in-memory state. In a multi-node deployment, exactly one node is the **leader** (elected via Kafka group coordination or ZooKeeper in older versions) and is the only one allowed to **write** to `_schemas`; followers forward writes to the leader and serve reads from their own caches. This makes the registry effectively stateless and horizontally scalable for reads, with Kafka providing durability, ordering, and replication. Replication factor of `_schemas` should be >=3 in production.
go deeper
Know the registry keeps its schemas in a Kafka topic called _schemas rather than its own database.
Explain single-partition + log-compaction and that nodes replay it into memory on startup.
Connect single-partition ordering to monotonic IDs and describe leader-only writes with follower forwarding.
Reason about durability (RF>=3), backup via topic mirroring, split-brain risks, and operational protection of _schemas.
## Design goal: no separate database The registry deliberately avoids running its own database. Instead it uses Kafka as its storage engine, treating a dedicated topic as a **durable, replicated commit log**. This reuses Kafka's replication and durability guarantees and keeps the registry process essentially stateless. ## The `_schemas` topic All registry state — schema definitions, subject/version mappings, per-subject compatibility config, and **delete tombstones** — is appended as records to a topic named **`_schemas`** (note the leading underscore, marking it internal). Key properties: - **Single partition.** A registry must assign **globally monotonic** schema IDs and a strict version order. A total order is only guaranteed within one partition, so `_schemas` has exactly **one partition**. This also means write throughput is modest, which is fine because schema registration is rare relative to message traffic. - **Log compaction (`cleanup.policy=compact`).** Kafka keeps at least the latest record per key and garbage-collects superseded ones. Deletes are represented as **tombstones** (null-value records). Compaction lets the log be replayed to reconstruct current state without growing forever. - **High replication factor.** In production the topic should have replication factor **>= 3** so losing a broker does not lose schemas. ## Startup and caching When a registry node boots, it **consumes `_schemas` from offset 0** to the end, materializing all schemas and subjects into an **in-memory store** (a `KafkaStore`). It then keeps tailing the topic to stay current. Because everything is in memory after replay, lookups by ID or subject are fast and the topic is not read per request. ## Leader/follower and writes With multiple registry nodes, only one is the **leader** (also called primary/master). Leadership is coordinated via Kafka's group protocol (or ZooKeeper in older releases). The leader is the **sole writer** to `_schemas`, which is what guarantees a single, conflict-free source of monotonic IDs. Follower nodes: - serve **read** requests from their own caches, and - **forward write** requests (register/delete/config changes) to the leader. This gives read scalability and a single serialization point for writes. ## Edge cases and operational notes - If `_schemas` is accidentally deleted or its replication is too low, schemas can be lost — it is a critical topic to protect (high RF, no manual deletion). - Because IDs are monotonic and derived from a single ordered log, you should **not** create multiple registries pointing at the **same** `_schemas` topic with conflicting leadership configs, or you risk split-brain ID assignment. A single logical registry cluster shares one `_schemas` topic. - Migrating or backing up the registry effectively means preserving the `_schemas` topic (e.g. via MirrorMaker) — there is no other store to back up. - The topic name is configurable (`kafkastore.topic`) but defaults to `_schemas`.
- Why must _schemas have exactly one partition?Kafka guarantees total ordering only within a single partition. The registry needs a strict order to assign globally monotonic schema IDs and consistent version numbers, so all writes go to one partition.
- In a multi-node registry, which nodes can write to _schemas?Only the elected leader writes; followers serve reads from their caches and forward registration/delete/config writes to the leader, ensuring a single source of monotonic IDs.
saying these in an interview costs you the question
- Saying the registry uses an external SQL database by default.
- Claiming _schemas has many partitions for throughput — it is single-partition by design.
- Saying any node can write schemas concurrently — only the leader writes.
- Forgetting that it is log-compacted (thinking it grows unbounded).