Several instances of a service boot together and each runs the schema-change step; what should the instances that do not apply it do?
answer
- everyone runs it, one applies it
- block rather than exit
- re-read after the lock frees
- bound the wait, release on death
basics
~20 sThey should block on the same exclusive lock, wait for the winner to finish, then re-read what is now applied and continue their own boot. Exiting, restarting, or skipping ahead to readiness are all worse than waiting.
solid answer
~50 sEvery instance runs the same step, so the step has to be safe to run concurrently. The usual mechanism is a database-held exclusive lock, taken before anything is inspected or applied. One instance gets it and applies the pending set; the rest block on the acquire call. When the winner commits and releases, each waiter wakes, re-reads what is now applied, finds nothing pending, and carries on to validating its mapping and becoming ready. The two bad alternatives are exiting so the platform restarts you, which turns a normal rollout into a crash-loop with backoff, and skipping the step to become ready early, which puts traffic on an instance before the schema exists. The wait must be bounded, and the lock should be tied to the session so a killed instance does not hold it forever.
go deeper
Recall the shape: identical instances start together, one takes a lock and applies the change, the others wait for it and then carry on. Nobody serves requests until their own boot finishes.
Explain where the lock lives — in the database, since that is the only thing all instances share — and why a waiter must re-read the applied state after it wakes rather than trusting what it saw earlier.
Show the operational consequences: crash-loop backoff if losers exit, health checks killing patient waiters, a lock left held by a killed process, and a bounded acquire so a stuck change fails with a message.
Weigh contention against a single designated applier. The leader removes the race but concentrates failure and still needs the rest to block, so decide which failure you would rather operate at fleet scale and standardise it.
## Why this is not solved by "only one instance should run it" When the runner is part of the deployable, every instance runs it, and a rollout starts several instances at once by design. You cannot arrange for only one of them to try: they are identical processes with identical configuration, started in parallel by a platform that does not know one of them is special. So the step must be **safe under concurrency**, and the interesting design question is what the instances that do not win should do while the winner works. The answer is: **wait, then continue**. Not exit, not skip, not apply anyway. ## The mechanism Mutual exclusion has to live where all the instances can see it, which means in the database itself rather than in any one process. Two shapes are common, and they differ in how they release: - **A session-scoped exclusive lock** offered by the engine, taken on a well-known key before the runner inspects or applies anything. Its virtue is that the engine drops it when the session ends, so an instance killed mid-boot does not leave the lock held. - **A lock row in a table**, claimed by an insert or an update that only one writer can win. Simple and portable, but it survives the process that took it, so it needs an owner identity, a heartbeat or an expiry, and an operator story for clearing a stale claim. Engines differ in what they offer here, and a data-access layer's own runner may pick either. Either way the sequence for every instance is the same: 1. Try to acquire the lock, with a bounded wait. 2. If acquired, determine what is pending, apply it in order, commit, release. 3. If the wait ended because the lock was released, re-read the applied state — do not assume it is still what you read before you started waiting — and apply whatever is genuinely left, which is normally nothing. 4. Continue boot: validate the mapping against the schema, then report ready. Step 3 is the one people skip. A waiter that acted on a snapshot taken *before* it blocked would try to re-apply the change the winner just made. ## Why waiting beats the alternatives | Behaviour on losing the race | What actually happens | |---|---| | Block on the lock, then continue | Instances become ready a few seconds apart; the rollout is simply as slow as the change | | Exit non-zero and let the platform restart | A crash-loop with backoff; restart counters climb, the platform may mark the deploy failed, and alerting fires on a normal rollout | | Skip the step and become ready | Requests reach an instance before the schema is in place — the failure the whole ordering exists to prevent | | Apply without taking the lock | Concurrent appliers race on the same objects: duplicate statements, deadlocks, an interrupted set | Waiting is also the behaviour that degrades gracefully. If the change is fast, nobody notices. If it is slow, the fleet becomes ready in a wave rather than failing. ## The single-leader variant Instead of every instance contending, some systems designate one applier: a one-shot job, or one instance elected through the platform. It removes the contention but not the coordination — the others must still not serve until the change is done, so they either wait on a signal or check the applied state themselves and wait for it to reach the expected point. It also concentrates failure: if the leader dies mid-change, nothing else picks the work up unless you build that, so the rollout must block rather than proceed on the assumption that the leader succeeded. ## What waiting still needs - **A bounded acquire.** An unbounded wait converts a stuck change into a fleet of processes hanging with no message. Fail with a clear error instead, and let the deploy stop. - **A startup allowance that fits.** The platform must tolerate an instance waiting on the lock without killing it as unhealthy; waiting instances are alive and not ready, which is exactly what a readiness signal is for. - **Release on death.** Whichever lock shape you use, decide what happens when the holder is killed at the wrong moment, and make the answer something an operator can act on rather than a permanently stuck deploy. - **Idempotent re-checks.** The waiter re-reading the applied state after the lock frees is what makes running the same step in ten processes harmless. The property to state in an interview is simple: the step is run by everyone, applied by one, and waited for by the rest — with a timeout on the waiting and a readiness signal that keeps traffic off all of them until it resolves.
- Why must a waiting instance re-read the applied state instead of using what it read before blocking?Because the winner changed exactly that state while it waited. Acting on the pre-wait snapshot means re-applying changes that are already in place — at best an error, at worst a second application of something not written to be repeated. Reading after acquiring the lock is what makes the step idempotent across processes.
- What goes wrong if the platform's health checks kill instances that are waiting on the lock?They restart, contend again, and can be killed again — a self-sustaining loop that never lets the change finish while the change is what they are waiting for. Waiting instances are alive but not ready, so the platform must distinguish the two and allow a startup window longer than the change.
saying these in an interview costs you the question
- Says only one instance runs the step, so no lock is needed
- Exits and relies on restarts to serialise the appliers
- Skips the schema step and becomes ready while another applies
- Uses a lock nobody releases when the holder is killed
- Applies from a state snapshot taken before waiting
- Waits on the lock forever with no timeout