Where does the contract store beside a cluster keep its own state, and why does that decide whether a restore of the brokers is usable?

## The question behind the question A continuity plan that says "restore the cluster" is describing one service. Producers depend on at least two. The second one - the **contract store**, the companion service a producer must reach before it may publish - is not a stateless proxy: it holds durable content that took months to accumulate, and that content has its own storage, its own owner and its own backup story. An interviewer asking this is checking whether you have ever looked past the brokers, because a plan that restores the cluster and nothing else hands you a healthy cluster no producer can use. ## The three places that state lives | placement | who owns its backup | what happens when the cluster is restored | |---|---|---| | a stream on the very cluster it serves | you do, by accident - it rides along with the cluster's own data | it returns with that data and not before, so the service must be started after the cluster is serving; if the stream was not in what you restored, the store comes up empty | | a separate database owned by another team | that team, on their schedule | you inherit their recovery point and their sequence; two numbers nobody compared become one plan the moment you depend on them | | derived from a checked-in definition the deployment applies | source control, and the deployment that reads it | it is reproduced rather than restored, which is the cheapest path - provided nothing was registered at runtime | The first row is the one worth dwelling on, because it is genuinely circular. The store's content is kept on the cluster the store gates writes to. Recover the cluster and the content is there; fail to recover that part of the cluster and the store starts with nothing, and every producer is refused at the first publish - with the brokers healthy and the dashboards green. ## What a usable restore actually requires 1. **Sequence.** Whatever holds the state must be serving before the service that reads it starts. Where the state is on the cluster, that means the cluster first, then the store, then producers - and the plan has to say so, because nothing enforces the order. 2. **Reachability at the configured address.** A restored or standby instance answering on a different address is, to every client, an outage. This is the step most often skipped when the switch goes to a standby cluster rather than back to the original. 3. **Content no older than the records.** If the store's state was recovered to an earlier point than the stream's, producers and readers meet payloads whose contracts the store has never held. The store's own recovery point is therefore a number in your plan, not the database team's private business. 4. **A real publish and a real read.** Process-is-running is not restored. The proof is one write and one read completing through the restored path. ## Why the "rebuild it from source" answer is only half true Deriving the contents from a checked-in definition is the cheapest posture and the easiest to rehearse: no backup, no ordering, no second team. It is honest only where the definition is complete. Two things break it: - **runtime registration** - where producers add entries as they start or as they ship a new payload shape, those entries exist only in the store, and no artifact knows about them; - **drift** - the definition in source control is what somebody last committed, which is not necessarily what the running store holds. If you claim this posture, the honest sentence is "we reproduce it from source, and we have checked that nothing registers at runtime", not "we can rebuild it". ## What varies Not every estate has this dependency at all. Some platforms ship a contract store beside the cluster; others have none, and contracts travel with the payload or in a shared build-time artifact. Some designs also keep cluster metadata in a separate coordination service with its own state, while others keep it inside the cluster - and where that separate service exists, everything above applies to it too, with a wider blast radius. The way to phrase the answer so it survives any platform is to ask the general question: *for every service a client must reach besides a broker, where does its state live and who restores it?* ## The failure mode in one line A plan that covers the brokers and not the companion beside them does not half-work. It restores a cluster that refuses every write, which is indistinguishable, from the producer's side, from not having restored anything.