A boot-time schema change stalls behind a lock and the rollout hangs; which waits do you bound, and how?

## Two waits wearing the same word "The migration is stuck on a lock" describes two unrelated situations, and the fix differs: - **The runner's mutual-exclusion lock.** Taken before anything is applied, so that exactly one process applies. Blocking here means somebody else is applying — a peer instance, a pipeline job, a colleague running it by hand — or somebody *was* and died holding the lock. - **The object locks the statements themselves need.** A change that alters or rebuilds a table needs the engine to grant it a lock on that object. Blocking here means live traffic, a long analytical read, or a transaction someone left open is holding an incompatible lock. The first is a coordination problem; the second is a contention problem with production traffic. Diagnosing one as the other wastes the outage. ## Bound each one separately 1. **An acquire timeout on the runner's lock.** After it expires the runner exits non-zero with a message naming the lock and, if the mechanism records it, its holder. That converts a silent hang into a failed deploy someone can act on. 2. **A lock-wait or statement timeout around the change itself.** Standard practice is to make the applying session give up quickly rather than queue: a blocked schema statement is not merely slow, it typically sits ahead of everything else queuing for the same object, so a change that waits ten minutes can stall reads and writes that would otherwise have succeeded. Failing after seconds and retrying later is usually better than waiting. 3. **A cap on the total apply.** Even with both above, a set of many changes can run long. A wall-clock budget for the whole run, checked between changes, keeps a rollout from disappearing into an unbounded apply. ## Order the deadlines from the inside out The timeouts nest, and getting the nesting wrong is the classic mistake: | Layer | Should expire | Why | |---|---|---| | Statement / object-lock wait | First | Frees the queue behind a blocked statement quickly | | Runner lock acquire | Second | Fails the instance with a clear coordination error | | Platform startup allowance | Last | Lets the runner report the real reason before anything is killed | | Rollout deadline | Last of all | The deploy fails after the instances have explained themselves | If the platform's startup allowance is the shortest, every failure mode looks identical from the outside — "the instance did not start in time" — and the process is killed at an arbitrary point in the change. ## Being killed mid-change is the expensive case - Engines differ in whether a schema change can be rolled back inside a transaction. Where it can, an interrupted change leaves nothing behind; where it cannot, part of the set is applied and the rest is not, and the next run has to cope with a state the change set did not anticipate. - A lock held in the killed session is released by the engine when that session ends; a lock recorded as a row in a table is not, and the next instance blocks on a claim whose owner no longer exists. - Restart backoff then re-runs the whole thing, so an interrupted change can be re-attempted while the previous attempt's effects are still being untangled. So the startup allowance is not a formality: it should be sized against the slowest realistic apply, and the runner's own timeouts should be strictly tighter, so the failure is always the runner's, reported in its own words. ## What to do when it does stall 1. **Read which timeout expired.** Acquire failure means look for another applier or a stale claim; statement-lock failure means look at what is holding the object. 2. **Identify the holder.** For the runner lock, whatever identity the mechanism records. For object locks, the engine's own view of sessions and what they are waiting on. 3. **Decide between waiting and stopping.** A change blocked behind live traffic on a busy object rarely gets a better window by waiting inside a deploy; stopping the rollout and running the change in a chosen window is often the right call. 4. **Clear stale state deliberately.** Releasing a lock claim whose owner is gone is an operator action with a checklist, not something the runner should decide on its own after a fixed interval — an over-eager auto-clear lets two appliers run at once, which is precisely what the lock existed to prevent. The interview point: name the two waits, bound both, nest them inside the platform's patience, and make the log say which one gave up.