skip to content

A build writes a feed credential to a file and removes it in a later step — why does the published image still carry it?

level: middleimportance: should knowfreq 50%

answer

  1. layers stack, they do not rewrite
  2. a removal is a marker, not an erase
  3. earlier bytes still transfer on pull
  4. the assembled view is not the artifact
  5. keep it out of every layer instead

basics

~20 s

Because layers stack rather than overwrite. The later step records that the path is hidden; the earlier layer that wrote the credential is unchanged, still transferred on every pull, and still readable by unpacking the layers.

solid answer

~50 s

Each build step contributes a layer holding the filesystem changes that step made, and the published image is the ordered stack of those layers. Removing a file in a later step does not reach back into an earlier one: it records a removal that hides the path when the layers are assembled into the view a running container sees. The earlier layer still contains the credential, is still pushed to the registry, is still pulled by everyone who pulls the image, and is readable by anyone who unpacks the layers one at a time instead of looking at the assembled result. That is why the standard check — start the image, list the path, see nothing — is the wrong check. The only reliable fix is to never let the value land in a layer.

code

pseudocode · 8 lines
pseudocode
step 1 -> layer A: write(feed_credential_file)
step 2 -> layer B: fetch_packages(token_file = feed_credential_file)
step 3 -> layer C: delete(feed_credential_file)

published image = [A, B, C]

assembled view  = C marks the path removed   -> nothing visible at runtime
layer A on disk = the credential, unchanged  -> pulled by everyone, readable offline

go deeper

for a junior

Recall the shape: layers are added, never edited, so deleting a file later hides it rather than removing it. A clean listing inside a running container is not evidence.

for a middle

Explain the removal marker and the difference between the assembled view and the stack of layers that was actually pushed, and say which one a puller can read.

for a senior

Show that you verify against the layers and the recorded build inputs, and that you treat every previously published image as still carrying the value regardless of what the current build does.

for a principal

The judgment a lead owns is whether builds can produce this at all — whether the scoped path is the default one teams get, and what stands between a build and publication.

## Why a later step cannot undo an earlier one A build is a sequence of steps, and each step contributes one **layer**: the set of filesystem changes that step made, captured at the point the step ends. The published image is that stack, in order, plus a document describing it. Layers are additive — a step does not edit the layers beneath it, it adds another one on top. So when a later step deletes a file that an earlier step created, the builder has only one way to express that: the new layer records a **removal marker** for the path. When the layers are assembled into the single filesystem view a running container sees, the marker wins and the path is absent. Nothing was erased; something was covered. ## What the published image therefore holds | Where you look | What you see | |---|---| | Inside a running container | Nothing at that path — the removal marker hides it | | The layers transferred on pull | Every layer, including the one that wrote the credential | | One layer unpacked on its own | The credential file, exactly as the step wrote it | | The recorded build inputs | The value too, if it arrived as a build argument or environment value | Unpacking layers individually is not exotic. It needs no special access and no running platform, only the image itself — which is precisely the artifact you published. ## Where the habit comes from The write-then-delete pattern comes from picturing the build as one filesystem being edited in place, the way a script edits a working directory: you create a temporary file, use it, and tidy up. A build is not that. It is a recorded sequence of snapshots, and the recording is the product. Anything true at the end of any step is preserved, not just what is true at the end of the last one. The same reasoning covers the variants people reach for next: - **Overwriting the file with zeroes in a later step** records a layer containing a zeroed file on top of a layer containing the credential. Both ship. - **Encoding the value before writing it** changes the bytes, not the fact that they are there and reversible. - **Clearing an environment value later** leaves the earlier setting in the recorded build inputs. - **Removing the file and then rebuilding with the same credential** publishes a fresh image that no longer shows the file, while every previously published image is unchanged. ## The fix, in order 1. **Stop the value landing in a layer at all.** Hand it to the one step that needs it through a step-scoped secret mount, so there is nothing to clean up afterwards. 2. **Check the step's other outputs.** A credential can re-enter the layer sideways — a generated configuration file, a cache directory, a lock file recording an authenticated URL. 3. **Verify against the layers, not the running container.** The check that means something inspects the layers and the recorded build inputs of the finished image. 4. **Treat anything already published as disclosed.** A corrected build produces a clean new image; it changes nothing about the images already pulled. ## The one nuance worth knowing Creating and removing the file **inside a single step** is different in kind, because the layer records the step's filesystem *as it ends* — a file that no longer exists at that moment is not in the layer. That is worth understanding so you can read an existing build correctly, but it is a fragile control rather than a design: the value can still be in the recorded build inputs if it arrived as an argument, in anything else the step wrote, and in the build's output stream. Scoping the value to the step is the mechanism; hoping nothing survived the step is not.

  • Does creating and removing the file inside one step leave it in that step's layer?
    No — a layer records the step's filesystem as the step ends, so a file that no longer exists then is not captured. It is still the wrong control rather than a fix: the value can remain in the recorded build inputs if it arrived as an argument, in anything else that step wrote, and in the build's output stream.
  • What is the check that actually tells you whether an image carries a credential?
    Inspect the image's layers and its recorded build inputs directly, rather than starting the image and listing a path. The running container shows the assembled view, which is exactly the view a removal marker was designed to change.
  • The build now uses a scoped mount, so is the old published image fine once the tag is reused?
    No. Republishing the tag adds a clean image; it does not alter the earlier one, which stays pullable by its content identity and stays in every mirror, cache and host that already fetched it. The published value has to stop being accepted.

Painting over a wall does not take back what is underneath. A later layer hides the file; the earlier layer already shipped the bytes, and anyone can look behind the paint.

saying these in an interview costs you the question

  • Says the file is gone because a running container does not show it
  • Believes the later removal edits the earlier layer
  • Thinks unpacking layers individually needs special access
  • Assumes removing the file also clears the recorded build inputs
  • Calls the incident closed once the removal step is added