Two engineers start an apply at the same time against the same shared IaC state record. What can go wrong, and why is locking the standard answer?

## Why a run is a critical section Strip an apply down and it is three steps: read the recorded state, make real changes to the world, write the record back. That is a read-modify-write over shared mutable data — the shape that always needs concurrency control. What makes it nastier than a database row is that the "modify" step has external side effects that cannot be rolled back by discarding a write. If the record write is lost, the infrastructure it described still exists. ## The interleaving, concretely Run A and run B start seconds apart against the same record. 1. A reads the record; B reads the record. Both hold the same snapshot. 2. A creates a load balancer and a security group, then writes the updated record. 3. B creates a queue and writes *its* record — derived from the snapshot it read in step 1, which knows nothing about A's two resources. The result: two real resources exist that no record mentions. They are billed, they are in the traffic path, and nothing manages them. The next run sees them missing from the record, concludes they were never created, and proposes to create duplicates — which may then fail on a name collision, leaving the pipeline red for reasons that have nothing to do with the change being deployed. A second, uglier variant: both runs decide to modify the same resource. The provider applies whatever arrives, in whatever order, sometimes rejecting the second call because the object is mid-update. Now the record describes an outcome that never existed, and the next diff proposes a change that makes no sense to the person reading it. ## What locking actually is Before any operation that may write the record, the tool acquires an exclusive lock — implemented in whatever system stores the record, using whatever primitive that system offers: a conditional write, a lease, a row, a stack-level operation guard on a hosted service. It holds the lock for the whole run and releases it at the end. A second run either blocks for a bounded time or fails immediately with "the state is locked", usually printing who holds it, since when, and from where. Two design points are worth naming in an interview. First, the lock lives with the record, not with the tool: it must be visible to every client that could write, which is why a record on a laptop cannot be safely shared. Second, read-only operations that never write the record do not need the lock, which is what lets a plan-style preview run alongside a queued apply. ## What a lock does not cover **Anything not going through this record.** A console edit, another team's pipeline, an autoscaler adjusting a capacity attribute — none of these touch your record, so none of them are excluded. Concurrency control here is about corrupting the record, not about being the only actor on the infrastructure. **Another configuration managing the same object.** If two records both believe they own a resource, both runs hold their own lock and fight anyway. That is an ownership defect, not a locking one. **Everything downstream of your lock's scope.** The lock is as coarse as the record: every resource tracked in it is serialised together, so an unrelated change waits behind a slow one. ## Stale locks A CI job killed mid-apply — cancelled build, evicted runner, expired credential — can leave a lock held by nobody. Every later run then fails to acquire it. Tools provide an escape hatch to release a lock by force, and it is genuinely dangerous: if the original process is still alive, forcing the lock recreates the exact race the lock existed to prevent. The correct sequence is to identify the holder from the lock metadata, confirm that process is really dead (check the CI run, the host, the user), and only then force the release — followed by a careful review of the record, because the interrupted run may have created resources it never recorded. ## Serialising above the tool Mature setups do not rely on the lock as the primary mechanism; they arrange for the collision not to happen. Pipelines use a concurrency group per environment so only one apply for a given record runs at a time, and queue the rest. Humans are kept out of applying to production entirely, so the pipeline is the only writer. The lock stays as the backstop that catches what process forgot — including the case where someone applies from a laptop during an incident. Stateless tools have no record to corrupt, but they are not exempt: two concurrent configuration runs against one host interleave file writes and service restarts, producing a half-applied configuration. The remedy is the same, moved up a level — serialise the runs.