Why do Dockerfiles so often chain many shell commands into a single RUN instruction with `&&`, and what goes wrong if you split them into separate RUN lines and clean up afterwards?

## How layers are produced Building an image runs instructions in order. `RUN`, `COPY` and `ADD` each execute in a temporary container started from the current image state, then commit the resulting filesystem diff as a new **layer**. Metadata-only instructions (`ENV`, `WORKDIR`, `LABEL`, `USER`, `EXPOSE`, `CMD`, `ENTRYPOINT`, `ARG`) change the image config, not the filesystem, so they no longer add filesystem layers. Layers are immutable and stacked by a union filesystem (overlay2 in practice). Reading a path walks down the stack until it is found. ## Why deletion doesn't shrink an image There is no way to remove content from a layer that is already committed. Deleting a file in layer N+1 writes a **whiteout** entry — a marker saying "this path is gone from here up". The union filesystem honours it, so the running container cannot see the file. But the manifest still references layer N, that layer is still pushed to the registry, still pulled by every node, and its content is still recoverable by anyone who unpacks the layers with `docker save` or a registry client. The consequences: 1. **Size.** Deleting a 300 MB toolchain in a later RUN saves nothing; it actually adds a small layer. 2. **Secrets.** A credential file written in one layer and deleted in a later one is still in the image, fully readable. This is why removing a leaked key needs a rebuild, not a follow-up deletion. ## What chaining fixes When create and delete happen inside the *same* `RUN`, only the net result is committed: ``` RUN apt-get update \ && apt-get install -y --no-install-recommends build-essential \ && rm -rf /var/lib/apt/lists/* ``` The package index existed only inside that temporary container; the committed diff never contains it. Equivalents in other ecosystems: `apk add --no-cache`, `pip install --no-cache-dir`, `npm ci && npm cache clean --force`, `yum install && yum clean all`. ## The apt-get update cache trap The second reason for chaining `update` with `install` is cache correctness. A `RUN`'s cache key is essentially the literal command string plus the parent layer. So `RUN apt-get update` on its own hits cache forever — its text never changes. Weeks later you edit the separate install line; the update layer is reused, and you install against a package index snapshot from weeks ago. Symptoms are ugly and confusing: `404 Not Found` on package URLs (the mirror rotated versions away), or a silently outdated, unpatched package. Chaining puts both into one instruction whose cache key changes whenever the package list changes. Pinning versions (`build-essential=12.9`) also forces invalidation and makes builds far more reproducible. ## Don't overcorrect "One giant RUN" is not the goal. Layers exist for cache reuse and for sharing between images: - Layers are content-addressed and **shared**: ten images on the same base pull those base layers once. - A stable early layer is reused across rebuilds. Collapsing everything into one instruction means any change re-runs everything. So group by *lifecycle*, not by minimising count. A good shape is: OS packages in one RUN (they change monthly), dependency install in another keyed on manifest files (weekly), application source and build last (per commit). Ordering by volatility this way is why `COPY package.json ./ && RUN npm ci` before `COPY . .` is standard: a source-only edit does not re-resolve dependencies. The old advice about a hard layer limit is obsolete — overlay2 supports well over a hundred layers — so readability and cache behaviour, not a magic number, should drive the split. Also note that many small layers add per-layer overhead in pull and extraction, so extremes hurt in both directions. ## Diagnosing `docker history <image>` shows each layer with its size and the instruction that produced it, which is the fastest way to find the fat layer. Tools that walk the layers can show which files each one added. When you find a 400 MB layer whose command ends in a `rm`, you have found a split that should have been a chain. In a multi-stage build, layer hygiene in the *builder* stage matters much less, since those layers never ship — but it still costs cache space and build time, so the discipline is worth keeping where it is cheap.