Across a fleet of services, how would you decide where the schema-change runner runs, and what may it assume about which versions are serving?

## The decision is about starts, not about deploys People compare placements by asking where the change belongs in the pipeline. The more useful question at fleet scale is **when do processes start?** An embedded runner does not run once per deploy; it runs once per process start. - A service the platform starts only during a rollout runs the runner a handful of times, all of them while somebody is watching. - A service that autoscales runs it whenever load rises, whenever a machine is drained, whenever a process crashes. The step then executes at 3am, unattended, on a schedule nobody chose. That second case is where an embedded runner stops paying. Not because the apply is dangerous — it is normally a no-op by then — but because it makes an unattended start depend on something that can block, contend for a lock, and refuse to become ready. ## Criteria worth writing down | Criterion | Points toward embedded | Points toward a separate step | |---|---|---| | How instances start | Deploys only | Autoscaling, unattended restarts | | Fleet size at rollout | A few instances | Many starting in parallel | | Length of the apply | Seconds | Minutes, or unpredictable | | Credentials | One account is acceptable | Serving must not hold schema rights | | Who should be paged | The service team, always | The deploy owner | | Environments | One or two, uniform | Many, easy to wire once | | Risk of being skipped | High, ad-hoc environments | Low, pipeline is mandatory | None of these is decisive alone. The pattern is that embedded placement suits small, simple, low-instance-count services and stops suiting anything that autoscales or has a change set long enough to be felt. ## Standardise, then allow exceptions At fleet scale the value of a consistent answer usually exceeds the value of the locally optimal one. An engineer paged at night should not have to know which of thirty services applies its schema from boot. A workable policy has three parts: 1. **One default placement** for services generated from the standard template. 2. **One mechanism** for mutual exclusion, so the failure and its remedy read the same everywhere. 3. **A named exception path** for the service that genuinely needs something else, with the reason recorded. The frequent compromise is to keep the change set inside the artifact — so it is version-locked to the code that needs it and cannot drift — while running it from a one-shot start of that same artifact rather than from the serving path. That keeps the coupling that makes embedded placement attractive and removes the property that makes it expensive. ## What the runner may assume while it runs This is the part candidates miss. Whatever the placement, at the moment the change is applied: - **The previous version is still serving.** Its instances are connected, holding pooled connections, and issuing statements against the objects being changed. - **The new version is not serving yet**, or only a first instance is. The runner therefore cannot assume the code that matches the new schema is the code taking traffic. - **Both may be live simultaneously** for the length of the rollout, and longer if the rollout pauses or reverses. The consequence for placement is direct: the change has to be one that the currently-serving version tolerates, because it will meet it before its own version does. Designing changes to satisfy that constraint is its own discipline and belongs elsewhere; what belongs here is the assumption itself, and the fact that moving the runner earlier in the deploy makes the window in which the old version meets the new schema *longer*, not shorter. ## Identity and blast radius A fleet-level policy should also fix who the runner connects as. Applying a change needs permission to alter the schema; serving requests does not. Embedded placement pushes toward one account that can do both, which means every serving process — and anything that reaches it — has the ability to alter the schema for the lifetime of the deployment. A separate step lets the schema-change identity exist only for the seconds the step runs. The detailed privilege design is a database-security question, but the placement decision determines whether that separation is even available to you. ## How to argue it in an interview State the criteria, pick a default, and be explicit about the exception path. Then show that you know the runner is not operating in a quiet system: it runs against a database that the previous version is actively using, on a schedule the platform may choose rather than the deploy, and with rights that should not outlive the change.