What is a golden (baked) machine image, and how do you decide what to bake into the image at build time versus what to configure when the instance boots?

## Baking versus frying Two ways exist to get a working server out of a base operating system. **Frying** launches a generic base image and does all the work at first boot: a boot-time script (cloud-init user data, for example) installs packages, writes configuration and pulls the application down. **Baking** does that work once, ahead of time, in a build pipeline: the result is a *golden image* — a versioned artifact containing the hardened OS, runtime, agents and usually the application, so the boot sequence is little more than starting processes. Most real estates land in between, and the interesting question is not "which one" but "where is the line", because the line determines what can fail at the worst possible moment. ## Why the line matters: what can fail at launch Everything you do at boot is work performed under time pressure, at the exact moment you are least able to tolerate failure — an autoscaling event during a traffic spike, or a replacement after an instance died. Anything fetched at boot is a live dependency of your ability to add capacity. If your boot script runs a package install, then your scale-out path depends on a public package repository being up, on the version you asked for still existing, and on egress to the internet being allowed. Baking converts those runtime dependencies into build-time dependencies, where a failure is a red build instead of an outage. Boot time is the second axis. A five-minute boot means five minutes between deciding you need capacity and having it, which sets a floor on how fast you can respond to load and how long a rolling replacement takes. ## The decision rule Bake it if it is: - **Slow** — compiling, large downloads, database or index warm-up. - **Version-sensitive** — anything where two hosts installing at different times would get different bits. - **Fetched from something you do not control** — third-party package repositories, language package registries, external artifact stores. - **The same everywhere** — the agent set, the hardening baseline, the runtime. Inject at boot if it is: - **Genuinely per-instance** — hostname, role, availability zone, cluster identity, an instance's shard assignment. - **Per-environment** — endpoints, sizing, feature flags, log levels. These belong in configuration read at start-up, not in a separate image per environment. - **A secret** — credentials fetched at start-up from a secret store, never baked, because an image is copied, shared across accounts and often readable by anyone who can launch it. The test that keeps you honest: *could the same image ID run in staging and in production?* If the answer is no, something environment-specific has been baked in, and you have lost the main benefit — that the artifact you tested is bit-for-bit the artifact you promoted. ## Phoenix servers A *phoenix server* is one deliberately destroyed and rebuilt from its image on a regular cadence rather than only when a change ships. The point is not tidiness; it is that the rebuild path is the recovery path, and an untested recovery path does not work. Rebuilding routinely also bounds how long a host can accumulate anything the image does not describe — an attacker's foothold, a manual fix, a leaked file descriptor, an unbounded log directory. Short instance lifetimes are a security and reliability property, not just an aesthetic one. ## The costs of baking Baking is not free, and a strong answer names the bill: - **Build latency.** A one-character change now requires an image build and a fleet roll. If that path takes an hour, people will be tempted to hot-fix by hand under pressure. - **Image sprawl and storage.** Every build is an artifact to store, copy between regions or registries, scan, and eventually garbage-collect — while keeping enough history that you can still launch the previous version. - **Patch latency across a layered base.** If a shared base image is rebuilt for a CVE, every downstream image must be rebuilt and every fleet rolled. That fan-out needs automation, or the base image quietly ages. - **Blast radius.** One base image used by the whole estate means a bad build can break the whole estate at once. Promote images through environments; do not let a build go straight to production because it is "just the base". ## Testing the image, not the host Because the image is the unit of change, it is also the unit of test: run the image in a throwaway environment and assert the things you care about — the service starts, the health endpoint answers, the agents report, the expected versions are installed — before promoting it. That is a cheaper and far more reliable gate than inspecting live production hosts, and it is only possible because the artifact is frozen.