You own an estate that includes stateless API fleets, a self-managed stateful database cluster, CI build agents, and a vendor appliance configurable only through its own console. How would you decide where to mandate immutable replacement and where to keep in-place management, and what does pushing immutability everywhere cost?

## Frame it as a per-workload decision, not a policy The weak answer is "immutable everywhere, it is best practice". The useful answer sorts the estate on axes that predict whether replacement is actually viable. **State gravity.** Does replacing this node move data? A stateless API instance carries nothing; replacing it costs a health check. A database member carries a working set that must be rebuilt or reattached, and the fleet must stay above quorum while it happens. The more data must move, the more replacement stops being a routine operation and becomes a planned one. **Replacement cost and time.** How long from launch to serving? A minute is routine and can happen on every deploy; forty minutes means replacement is a scheduled activity and there is a real ceiling on how fast you can respond to load or to a CVE. **Cardinality and divergence exposure.** Three hosts can be inspected by a person. Three hundred cannot, and any process that relies on humans keeping them consistent has already failed. Large fleets are where immutability pays most. **External constraints.** Licensing tied to a host identity, a vendor appliance whose only interface is its console, hardware, or a compliance process that requires change approval per host — these can veto replacement regardless of what you would prefer. ## Applying it to this estate **Stateless API fleets** — hard mandate. Every change ships as a new image; no interactive access to production instances for changes; replacement is the only path. This is where the model's benefits are unqualified. **CI build agents** — hard mandate, and arguably the highest-value case. An agent that has been running for months carries toolchains, caches and leftovers from every job it has run, so builds start depending on the agent that happened to pick them up, and "works on agent 4" becomes a real failure mode. Ephemeral agents built from an image also contain the blast radius of untrusted build code. The tension is cache warmth, which you solve with an external cache rather than by keeping agents alive. **Self-managed stateful cluster** — immutable *nodes*, deliberate replacement. Put the data on storage with a lifecycle independent of the instance so a member can be replaced without a full rebuild, and replace one member at a time, respecting quorum and rebalancing, verifying replication health between steps. The rate is minutes-to-hours per node, not seconds, so this workload is on a different cadence rather than a different philosophy. **Vendor appliance** — no immutability available. Do not pretend otherwise; put compensating controls in place. Export its configuration on a schedule into version control so changes are diffable and attributable, rehearse a restore from that export, restrict who can change it, and record it on an explicit exception register with a named owner and a review date. An honest exception you can audit beats a policy everyone quietly ignores. ## The costs of pushing immutability everywhere A principal-level answer names the bill rather than only the benefits. - **A build platform to own.** Image pipelines, a registry or image store, scanning, promotion, retention and garbage collection, plus the base-image rebuild fan-out whenever a CVE lands. That is a product with an owner, not a one-off project. - **Lead time on trivial changes.** If a one-line fix takes an hour of build and roll, people will hot-fix by hand during an incident. Either the pipeline is fast enough to be the emergency path, or you write down a sanctioned break-glass procedure that ends with a rebuild. What you must not do is have a rule that reality routinely breaks. - **Storage and sprawl.** Every build is an artifact to store, copy across regions or accounts, and eventually delete — while retaining enough history to remain able to roll back. - **Concentrated blast radius.** A shared base image is a single point of failure for the whole estate: one bad build can break everything at once. That argues for promotion through environments and staged roll-outs of base-image changes, not for abandoning the shared base. - **Capacity during replacement.** Replacing rather than patching generally means running old and new simultaneously, so there is a headroom and cost implication, and a quota implication in a constrained account or region. - **Debugging changes shape.** You can no longer poke the sick host, so you need off-host logs and metrics, the ability to launch the exact image locally or in isolation, and a policy on quarantining a bad instance for inspection instead of terminating it immediately. ## The principle to state out loud Enforce immutability where replacement is cheap and state-free; where it is not, the goal does not change — you still want the host's contents to be derived from a reviewed, reproducible source. What changes is the mechanism: reproducible builds, pinned inputs, exported configuration under review, and detectable divergence, carried as a named exception with an owner rather than as an oversight.