Under BuildKit, why is there no intermediate image to run after a failed build step, and how do you get a shell at that point?

## Why the old trick stopped working The classic builder built an image by running one container per instruction and committing it. Its output looked like this: ``` Step 12/18 : RUN go build -o /out/gateway ./cmd/gateway ---> Running in 7c1f2a ---> 9b4e11 ``` Each `---> <id>` was a real image, and after a failure the id printed by the last *successful* step was still sitting in `docker images` (as a dangling image). The universal debugging move was `docker run --rm -it 9b4e11 sh`, and from there you re-ran the failing command by hand. BuildKit — the default builder in Docker Engine 23.0 and later — does not work that way. It resolves the Dockerfile into a graph of steps, executes independent steps concurrently, and stores each completed step's filesystem as a content-addressed snapshot in the **builder's cache**, which is a separate store from the engine's images. An image is produced only by an *export* at the end of the build, for the stage you asked for. When the build fails, no export happens, so nothing lands in `docker images`, there are no dangling intermediates to list, and there is no id to run. The snapshot of the step before the failure exists, but it is not addressable as an image. This is a real ergonomics loss that BuildKit trades for concurrency, better caching and secret handling, and it is why interviewers ask the question: a candidate who still answers "run the intermediate image" has not built an image since 2022. ## The replacement: --target a probe stage `--target <stage>` tells BuildKit to build only up to the named stage and export *that* as the image, skipping every stage that stage does not depend on. Stage results **are** exportable images, so a stage boundary is the supported way to get a runnable artefact from the middle of a build. So the technique is: cut a stage boundary immediately before the failing instruction. ```dockerfile FROM golang:1.24-alpine AS deps WORKDIR /src COPY go.mod go.sum ./ RUN go mod download FROM deps AS prebuild COPY . . FROM prebuild AS build RUN CGO_ENABLED=0 go build -trimpath -o /out/gateway ./cmd/gateway FROM scratch COPY --from=build /out/gateway /gateway ENTRYPOINT ["/gateway"] ``` `docker build --target prebuild -t gateway-probe .` gives you an image holding everything the failing `go build` could see. `docker run --rm -it gateway-probe sh` puts you in it. Now paste the command: ``` /src # CGO_ENABLED=0 go build -trimpath -o /out/gateway ./cmd/gateway ``` and you get the error interactively, with a shell to explore: `ls`, `env`, `cat go.mod`, run it again with more verbosity, edit a file and retry. That loop is worth far more than another full build. If the Dockerfile has no boundary where you need one, add one temporarily — a `FROM <previous stage> AS probe` line costs nothing and is deleted afterwards — or simply comment out the failing instruction and everything below it and build without `--target` at all. Both produce the same thing. ## Details that matter in the room **The probe must be *before* the failure, not the enclosing stage.** `--target build` in the file above still runs the failing `go build`; you want the stage that ends just above it. **Watch the shape of the final stage.** A Go binary shipped `FROM scratch` has no shell at all, so "just run the image" was never available here even on a success; the thing you debug is the builder stage, which is a full distribution image with a shell. **Running the base image alone is a weaker version of the same idea.** `docker run --rm -it golang:1.24-alpine sh` and typing the commands by hand does reproduce some failures, but you are missing every `COPY`, `ENV`, `ARG`, `WORKDIR` and `USER` the build had applied. Differences between your hand-run and the build's environment are exactly where these bugs hide, so prefer a `--target` probe, which reproduces them for free. **Non-interactive context.** Even inside the probe, remember the build ran with no TTY and as whatever `USER` was in effect; a command that behaves differently under a terminal (a package manager prompting, a tool colourising output) can pass by hand and fail in the build. Re-run with the same flags the Dockerfile used before concluding the environment is identical. **There is an experimental shortcut.** buildx ships a debug monitor that can drop you into the container of the step that failed, gated behind the `BUILDX_EXPERIMENTAL=1` environment variable. It is genuinely useful, but its command surface has moved between buildx versions, so know it exists and do not build a team runbook on it. The `--target` probe works on every BuildKit version and in CI.