Why is a nightly batch job's image over a gigabyte when the compiled artifact it runs is 40 MB?

juniormust knowfreq 72%

answer

the image is the build environment
what the run needs, not the build
toolchain, headers, source, caches all ship
later deletion hides, does not reclaim
clean final stage takes only the artifact

basics

~20 s

A single-stage build ships everything the build needed, not only what the run needs: the compiler toolchain, development headers, the whole source tree, intermediate output and package-manager caches all stay in the image's layers beside the 40 MB artifact.

solid answer

~50 s

The image is the build environment, frozen. In a single-stage build every step appends to the same chain of layers, so the compiler, the headers it needed, the full source tree, intermediate object files and everything the package manager downloaded are still there when the build finishes — and every host that runs the job pulls all of it. Deleting them in a later step does not help: earlier layers are immutable, so the bytes stay in the image and the later step only records that the files are gone. The fix is structural, not cosmetic. Build in a toolchain stage that has the compiler, then start a clean final stage and copy in only the finished artifact and its config file. What ships is then tens of megabytes and contains no compiler at all.

The image is the build environment, frozen. In a single-stage build every step appends to the same chain of layers, so the compiler, the headers it needed, the full source tree, intermediate object files and everything the package manager downloaded are still there when the build finishes — and every host that runs the job pulls all of it. Deleting them in a later step does not help: earlier layers are immutable, so the bytes stay in the image and the later step only records that the files are gone. The fix is structural, not cosmetic. Build in a `toolchain stage` that has the compiler, then start a clean `final stage` and copy in only the finished artifact and its config file. What ships is then tens of megabytes and contains no compiler at all.

code

pseudocode · 12 lines

pseudocode

stage toolchain:
  base    = image carrying compiler and build tooling
  inputs  = source tree, dependency manifest
  compile -> /out/app            # ~1.2 GB of tools, headers and objects live here

stage final:
  base    = small run-time base, no compiler, no package manager
  take from stage toolchain: /out/app -> /app
  take from build context:   app.conf -> /etc/app.conf
  entry   = /app --config /etc/app.conf

# shipped layers = final stage only; stage toolchain is not its ancestor

go deeper

for a junior

Remember the one-line cause: a single-stage build keeps the build environment, so the compiler, headers, source and caches ship with the artifact. Naming three of those items and proposing a clean final stage is a full answer here.

for a middle

Explain the mechanics: layers are append-only and immutable, so a later deletion only records that files are gone while the bytes stay in the earlier layer and are still pulled. Then describe the toolchain-stage-plus-final-stage shape concretely.

for a senior

Show that you price the gigabyte properly — repeated transfer on every scale-up and replacement, an intruder handed a compiler and a package manager, the source tree readable by anyone who can pull, and a long list of components you must answer for in advisories.

for a principal

The judgment is where to enforce this. Decide whether shipping a build environment is a defect the organisation blocks by policy or leaves to each team, and weigh that against the cost of maintaining a standard final-stage base that every service can actually run on.

Why a build's leftovers end up in the shipped image

An image is a stack of read-only layers, and in a single-stage build every step of the build appends to that same stack. The step that installs a compiler, the step that downloads dependencies, the step that unpacks the source and the step that compiles all write into the image that is eventually pushed and pulled. Nothing in that model distinguishes needed to produce the artifact from needed to run it — the builder cannot tell which files are which, so it keeps them all.

For a compiled workload the split is lopsided. What the build needed is typically:

The toolchain — compiler, linker, build tool, and the package manager used to install them. Usually the single largest item, often several hundred megabytes on its own.
Development headers and static libraries pulled in to satisfy the compile, which the finished artifact never reads.
The full source tree, plus whatever local files the build context happened to carry in with it.
Intermediate output — object files, generated code, test fixtures, coverage data.
Package-manager caches and index metadata, downloaded once and never read again.
Anything used only to authenticate to a dependency source while building.

What the run needs is one 40 MB artifact and a configuration file.

Why deleting at the end reclaims nothing

Layers are immutable once written. A removal performed in a later step of the same chain records a deletion marker in a new layer: a running container no longer sees the files, but the bytes are still in the earlier layer, that layer is still part of the image, and it is still stored in the registry and still transferred to every host that pulls. So install the toolchain, build, then uninstall the toolchain changes what the process sees and not what is stored or transferred. The same fact is why a credential written into a layer is not made safe by deleting it in a later step.

The shape that actually works

A toolchain stage starts from a base that carries the compiler and build tooling, receives the source, and produces the artifact at a known path.
A final stage starts fresh from a small base and copies in only that artifact, plus the run-time data it genuinely needs — a config file, a certificate store, a timezone database.
The shipped image is the final stage alone. The toolchain stage's layers are not ancestors of it, so they are not part of what is pushed or pulled.

	Single stage	Toolchain stage + clean final stage
What ships	build environment plus artifact	artifact plus its run-time data
Typical size here	over a gigabyte	tens of megabytes
Transferred to every host	all of it, on every pull	the artifact and a small base
Tools inside the boundary	compiler, package manager, shell	only what the small base carries
Components to patch and answer for	every build package	the run-time set only

Why size is not only a size problem

Transfer cost. The image is pulled on every host that has not seen it — on each scale-up, each replacement, each new node. A gigabyte of toolchain is paid for again and again.
Attack surface. A compiler and a package manager inside the boundary turn a foothold into tooling: an intruder can fetch, build and run whatever they like without bringing anything in.
Disclosure. The source tree is in the image. Anyone who can pull the image reads the code.
Patch burden. Every package present is a component that shows up in inventories and advisories, whether or not the workload uses it.
Build-only credentials. Anything the build authenticated with is in the build environment; if the build environment is what ships, that value ships too.

The usual result of the change, for a workload like this one, is an image somewhere in the tens of megabytes: the artifact, the config file, and a base that carries the handful of files the artifact actually reads.

Where this question stops

This leaf owns the shape — build in one stage, ship another. What the final stage starts from — a full distribution against a small or empty base, and what you give up with each — is a separate decision. So is diagnosing the result once it has no shell in it, how a build-time credential should be supplied and rotated, and how to order steps so the next build reuses more of the last one. Those are their own subjects; this one is simply that the build environment and the run-time environment are not the same image, and ought not to be.

## Why a build's leftovers end up in the shipped image An image is a stack of read-only layers, and in a **single-stage build** every step of the build appends to that same stack. The step that installs a compiler, the step that downloads dependencies, the step that unpacks the source and the step that compiles all write into the image that is eventually pushed and pulled. Nothing in that model distinguishes *needed to produce the artifact* from *needed to run it* — the builder cannot tell which files are which, so it keeps them all. For a compiled workload the split is lopsided. What the build needed is typically: - **The toolchain** — compiler, linker, build tool, and the package manager used to install them. Usually the single largest item, often several hundred megabytes on its own. - **Development headers and static libraries** pulled in to satisfy the compile, which the finished artifact never reads. - **The full source tree**, plus whatever local files the build context happened to carry in with it. - **Intermediate output** — object files, generated code, test fixtures, coverage data. - **Package-manager caches and index metadata**, downloaded once and never read again. - **Anything used only to authenticate to a dependency source while building.** What the run needs is one 40 MB artifact and a configuration file. ## Why deleting at the end reclaims nothing Layers are immutable once written. A removal performed in a later step of the same chain records a deletion marker in a new layer: a running container no longer sees the files, but the bytes are still in the earlier layer, that layer is still part of the image, and it is still stored in the registry and still transferred to every host that pulls. So *install the toolchain, build, then uninstall the toolchain* changes what the process sees and not what is stored or transferred. The same fact is why a credential written into a layer is not made safe by deleting it in a later step. ## The shape that actually works 1. A **toolchain stage** starts from a base that carries the compiler and build tooling, receives the source, and produces the artifact at a known path. 2. A **final stage** starts fresh from a small base and copies in only that artifact, plus the run-time data it genuinely needs — a config file, a certificate store, a timezone database. 3. The shipped image is the final stage alone. The toolchain stage's layers are not ancestors of it, so they are not part of what is pushed or pulled. | | Single stage | Toolchain stage + clean final stage | |---|---|---| | What ships | build environment plus artifact | artifact plus its run-time data | | Typical size here | over a gigabyte | tens of megabytes | | Transferred to every host | all of it, on every pull | the artifact and a small base | | Tools inside the boundary | compiler, package manager, shell | only what the small base carries | | Components to patch and answer for | every build package | the run-time set only | ## Why size is not only a size problem - **Transfer cost.** The image is pulled on every host that has not seen it — on each scale-up, each replacement, each new node. A gigabyte of toolchain is paid for again and again. - **Attack surface.** A compiler and a package manager inside the boundary turn a foothold into tooling: an intruder can fetch, build and run whatever they like without bringing anything in. - **Disclosure.** The source tree is in the image. Anyone who can pull the image reads the code. - **Patch burden.** Every package present is a component that shows up in inventories and advisories, whether or not the workload uses it. - **Build-only credentials.** Anything the build authenticated with is in the build environment; if the build environment is what ships, that value ships too. The usual result of the change, for a workload like this one, is an image somewhere in the tens of megabytes: the artifact, the config file, and a base that carries the handful of files the artifact actually reads. ## Where this question stops This leaf owns the *shape* — build in one stage, ship another. **What the final stage starts from** — a full distribution against a small or empty base, and what you give up with each — is a separate decision. So is **diagnosing the result** once it has no shell in it, **how a build-time credential should be supplied and rotated**, and **how to order steps so the next build reuses more of the last one**. Those are their own subjects; this one is simply that the build environment and the run-time environment are not the same image, and ought not to be.

The final image is now 60 MB but the build takes just as long — why?
Because the same work still happens. The toolchain stage still installs tools, resolves dependencies and compiles; the change only decides what is kept afterwards. Splitting stages is a size and surface fix, not a build-time fix — build time is decided by what the builder can reuse from the previous run.
Of everything the single-stage image carried, which items were risky rather than merely heavy?
The source tree, because anyone who can pull the image reads it. Any credential the build used, because it is in the build environment that shipped. And the compiler plus package manager, because they give anyone who gets a foothold the means to fetch, build and run new code without carrying anything in.

Posting a finished chair to a customer inside the whole workshop — saw, offcuts, blueprints and all — because the chair was built there. Sweeping the sawdust under a later layer of packing paper does not make the parcel lighter.

saying these in an interview costs you the question

Says a later cleanup step removes the toolchain from the finished image
Blames the base image when the toolchain and source tree are the bulk
Treats image size as a disk concern only, not pull time or attack surface
Claims the compiler must ship because the artifact was compiled
Assumes shipping the source tree is harmless because the artifact is small

Why is a nightly batch job's image over a gigabyte when the compiled artifact it runs is 40 MB?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why a build's leftovers end up in the shipped image

Why deleting at the end reclaims nothing

The shape that actually works

Why size is not only a size problem

Where this question stops

▾common follow-ups2 questions

▾analogy~20 s

▾red flags5 flags

Why a build's leftovers end up in the shipped image

Why deleting at the end reclaims nothing

The shape that actually works

Why size is not only a size problem

Where this question stops

answer

go deeper