A CI/CD pipeline builds the application from source separately for the dev, staging and production deployments. What does the "build once, deploy many" principle say to do instead, and what risk does rebuilding per environment introduce?
answer
- the evidence chain from test to production
- one build, many deployments
- same commit, two different artifacts
- configuration injected, never compiled in
- rollback redeploys, it never rebuilds
basics
~20 sBuild once, deploy many means the pipeline produces a single versioned artifact and moves that exact artifact through every environment. Rebuilding per environment means production runs bits nobody tested, so the staging result no longer proves anything.
solid answer
~50 sThe pipeline should compile and package exactly once, publish an immutable, versioned artifact — a container image, a jar, a wheel, a signed binary — and have every later stage deploy *that* artifact by its identity. Deploy jobs take an artifact identity as input; they never invoke the compiler. Rebuilding per environment breaks the evidence chain: two builds of the same commit can differ because dependency ranges, base images, system packages and the runner's toolchain all resolve at build time, so what production runs is not what staging approved. It also wrecks rollback — you can no longer redeploy the thing that was working, only rebuild it, which takes a full build cycle during an outage and may not reproduce the same bits. The precondition is that per-environment configuration lives outside the artifact and is injected at deploy time.
go deeper
Be ready to state the principle plainly — one build produces one artifact that every environment deploys — and to name the risk in a sentence: what you tested is not what ships.
Explain the mechanics of drift: dependency ranges, base images and runner toolchains all resolve at build time, so two builds of one commit can differ. Show where configuration enters instead.
Show the operational consequences you have lived: rollback that becomes a rebuild during an incident, an unanswerable "what is running right now", and a diagnosis with two variables instead of one.
Own the tradeoff between artifact storage cost and recovery time, and set the organisational rule — deploy stages carry no build toolchain — plus how you would migrate teams whose configuration is currently compiled in.
## What the principle actually says "Build once, deploy many" puts exactly one place in the delivery path where source becomes a runnable thing: the build stage. That stage compiles, packages and publishes a **versioned, immutable artifact** — a container image, a jar or war, a Python wheel, a tarball, a signed binary — and records its identity. Every later stage, in every environment, consumes that artifact by that identity. Deploy jobs are dumb: they take an artifact identity plus environment configuration and put the two together. They never check out source, never call the compiler, never run a packaging step. ## Why two builds of the same commit are not the same artifact The instinct behind rebuilding is "same commit, same output". That is a property you have to engineer and verify, not one you get for free. Things that routinely drift between two builds of one commit: - **Dependency resolution.** A manifest that permits ranges (`^1.4.0`, `1.4.+`, an unpinned plugin) resolves against a package registry *at build time*. A lockfile narrows this for the application's direct graph, but build plugins, toolchains and container base images are frequently outside it. - **Base images and system packages.** `FROM node:22` or an `apt-get install`/`apk add` line resolves to whatever the upstream index publishes today. - **The build machine.** The hosted runner image is upgraded on its own schedule; the second build may compile with a different compiler patch level or link a different system library. - **Build metadata and ordering.** Timestamps, hostname, locale, file ordering in archives, minifier or code-generator nondeterminism. So the artifact production runs is a *different* artifact from the one your staging tests approved — and nobody can say how it differs, because nothing compared them. ## What that costs **Test evidence.** Every pre-production test you ran was evidence about an artifact you then discarded. A green staging run tells you the staging build works. **Diagnosis.** "Works in staging, fails in production" now has two candidate causes — the environment *and* the artifact — instead of one, and you cannot cheaply eliminate either. **Rollback.** Rollback means running the thing that was working ten minutes ago. If that thing exists only as source, rollback becomes a rebuild: minutes to tens of minutes of build time exactly when you have none, and it can fail outright because an upstream dependency was yanked, a base image tag moved, or a build secret expired. **Auditability.** "What is running in production?" should be answerable with an artifact identity, not with a branch name and a hope. ## The shape of the pipeline instead ``` build: checkout -> test -> package -> publish emits: app@<immutable-id> deploy dev: app@<immutable-id> + dev config deploy stg: app@<immutable-id> + staging config deploy prod: app@<immutable-id> + production config ``` The value that flows down the pipeline is the artifact identity, not the source ref. Everything downstream is parameterised by it. ## What must be true for it to work - **Configuration and secrets live outside the artifact** and are supplied at deploy or start-up. An artifact with a hostname compiled into it cannot be promoted; it can only be rebuilt. - **The artifact store outlives the pipeline run**, with retention that covers your rollback horizon. - **Environments reference an identity that cannot be reassigned** — a content digest or a version that is never republished — so the reference means one specific set of bytes forever. - **The same artifact must be able to run at every size.** Replica counts, memory limits and pool sizes are environment configuration, not build inputs. ## The usual pushback *"Our build is deterministic, so rebuilding is harmless."* Sometimes true, and worth having — reproducibility is a real engineering goal, verified by rebuilding and comparing hashes rather than assumed. But even a perfectly reproducible rebuild costs the wall-clock time you do not have during an incident, and it still gives you no artifact to point at when someone asks what production ran last Tuesday. *"Each environment needs different settings, so each needs its own build."* This confuses code with configuration. The fix is to move the settings out, which is the same work you need for build-once anyway.
- If the build were fully reproducible, would rebuilding per environment become acceptable?It would remove the correctness argument but not the operational one. You would still spend a full build cycle to roll back, still have no stored artifact to identify what production ran, and still depend on every upstream source being reachable at rollback time. Reproducibility is worth having as a verification tool — rebuild and compare hashes — rather than as a licence to rebuild in the deploy path.
- What does a deploy job actually take as its input in a build-once pipeline?An artifact identity plus the target environment. The identity must be immutable — a content digest or a version that is never republished — so the job resolves to exactly one set of bytes. Everything that differs per environment (endpoints, credentials, sizing) arrives as configuration alongside it. The job needs no source checkout and no build toolchain at all.
- Where does the artifact's identity come from if the pipeline only builds once?The build stamps it: a version derived from the commit or tag being built, plus the content-addressed identity the artifact store returns on publish. The pipeline then passes that identity to every downstream stage as an output, and the running application reports the same value, so a health endpoint and the deployment record agree on what is live.
It is the difference between serving the dish you cooked and inspected, and cooking a second one from the same recipe for the customer. The recipe matches; the plate that goes out was never checked.
saying these in an interview costs you the question
- The same commit always produces the same artifact
- Rebuilding is fine because the tests run again
- Rollback just means rebuilding the previous tag
- Each environment needs its own build to get its settings
- Containers make builds reproducible automatically