In infrastructure operations, what is the difference between mutable and immutable infrastructure, and what happens to a running server under each model when a configuration or package change has to ship?

## The two models, one sentence each **Mutable infrastructure** treats a server as a long-lived thing you keep improving: you provision it once and then patch, upgrade, reconfigure and hot-fix it for months or years. **Immutable infrastructure** treats a server as a disposable rendering of a versioned artifact: when anything about it must change, you build a new artifact, launch new servers from it, and throw the old ones away. An *artifact* here means a machine image (a VM image such as an AWS AMI) or a container image — a frozen, addressable blob of bits with an identifier, produced by a build. ## What actually happens when a change ships In the mutable model the change lands as a *delta applied to a running machine*. Someone connects over SSH, or a configuration-management agent applies a manifest, and the package is installed or the config file rewritten. The machine's state afterwards is a function of its whole history: the base image chosen the day it was created, every package version that happened to be current on the day of each run, and every manual fix anyone ever made. Two machines with the same declared configuration can be materially different because they have different histories. In the immutable model the change lands *in source*. A build produces a new image; the image is tested and promoted; new instances are launched from it; the old instances are terminated. Nothing on a live host is edited — no package installs, no config rewrites, no interactive fixes. The consequence that matters is this: a host's configuration is fully described by one identifier, the image it booted from. Comparing two hosts becomes comparing two strings. ## Why interviewers care — snowflakes and drift A *snowflake server* is a host nobody can reproduce: it carries an undocumented kernel setting from a 3am incident, a library installed by hand, a config file edited during an outage and never written back to the repository. Snowflakes make three things expensive. Scaling out is risky, because the new machine your automation builds is not the same as the ones already serving. Debugging has to start with "which host was it?". And disaster recovery is a bet that the automation can rebuild what production actually is, rather than what it was declared to be. Immutability attacks this at the root: the only way to change a host is to change the artifact, so the artifact becomes the single source of truth and reviewing the code that produces it becomes the change-control process. ## Cattle, not pets The slogan says pets are named, nursed back to health when sick, and irreplaceable; cattle are numbered, interchangeable, and replaced rather than treated. Treating servers as cattle is not an attitude, it is a set of prerequisites: - No irreplaceable local state on the host — data lives in managed stores or volumes with a lifecycle independent of the instance. - Logs and metrics are shipped off-host, so terminating an instance does not destroy the evidence you need to debug it. - A replacement can be launched with no human steps, and health checks detect and replace bad instances automatically. If killing a random production instance would lose data or require a person, you have pets regardless of what your tooling is called. ## What the immutable model requires you to build A build pipeline that produces versioned images and a registry to keep them in; configuration injected at launch (instance metadata, environment variables, a parameter or secret store) so that *one* image can serve every environment; state deliberately externalised; and automated, well-tested replacement, because from now on every change is a replacement. That last point is the honest cost: you can no longer ship a one-line fix by editing a file, so your image build and roll-out must be fast enough to be the emergency path too. ## Where the model does not apply cleanly Stateful nodes such as database or broker members, hosts whose rebuild takes hours, and vendor appliances that can only be configured through their own console are all poor candidates for wholesale replacement. Immutability is a tradeoff, not a moral position. It is also layered: it is perfectly coherent to replace containers on every deploy while patching the hosts underneath them in place — just be explicit about which layer you are describing. ## Confusions worth clearing up "Immutable" does not mean the infrastructure never changes; it changes constantly, by replacement rather than by edit. Using containers does not by itself make you immutable — attaching to a running container to fix it puts you straight back in the mutable model. And rollback differs in kind, not just in speed: in the mutable model you must run the inverse of a change, and many changes have no inverse, while in the immutable model you launch an artifact that still exists exactly as it was.