Your admission check verified an image by tag; what makes the node pull exactly the bytes that were verified?
answer
- A tag is a name, a digest is the bytes
- Two reads of the registry, one window
- Rewrite the reference or reject tags
- Mirrors may resolve a tag differently
- The pod outlives its admission check
basics
~20 sNothing, unless the verifier rewrites the reference to the digest it checked. A tag is a mutable pointer, so the registry can resolve it to different bytes between the admission check and the node's pull.
solid answer
~50 sA tag is a name, not content. If the pod spec still says `app:1.4` after admission, the verifier resolved that tag, checked a signature over the digest it got, and then handed the workload back with the mutable name intact — the kubelet resolves it again, possibly to something else, and possibly through a different mirror with different credentials. Both Kubernetes-side implementations can close this: Kyverno's `verifyImages` has a `mutateDigest` setting that rewrites the reference to the digest it verified, and a `verifyDigest` setting that can require references be digests already; Sigstore's policy-controller can likewise resolve the tag to the digest it admitted. Client-side verification, such as running Notation in a pipeline, verifies nothing about what the cluster later pulls unless the digest it verified is what gets deployed. The durable fix is that whatever executes is named by digest, either because you write digests or because the verifier substitutes them.
go deeper
Know the core fact: a tag can be repointed at different content, a digest cannot. Verifying a tag says nothing about what gets pulled a minute later.
Explain the two registry reads and the window between them, and name the mechanism that closes it: the verifier rewriting the reference to the digest it checked, or a rule that rejects non-digest references.
Show you audit the seams — which mutating webhooks touch containers after yours, whether nodes pull through a mirror the verifier never reads, and what happens on a re-pull weeks later.
Frame it as an invariant to hold across the estate: what executes is named by content. Then decide whether you buy that with digest-only manifests, which pushes cost onto product teams, or verifier substitution, which concentrates it in the platform.
## The gap Verification and execution are two separate reads of the registry. Between them sits a window, and whether anything can move in that window depends entirely on how the artifact is named. A digest is content-addressed: `sha256:9f2a...` *is* the bytes, so a second read either returns those bytes or fails. A tag is an ordinary mutable pointer in a registry's namespace. If the admission check resolves `app:1.4`, verifies a signature over the digest that resolution returned, and then admits a pod whose spec still says `app:1.4`, the kubelet performs its own resolution later. This is a time-of-check-to-time-of-use gap, and the interesting thing about it is that the verifier passed honestly — it really did verify a signed artifact. It just did not verify *the* artifact that ran. ## Who can move the tag The threat here is not usually an anonymous outsider. It is whoever can push to that repository: a compromised CI credential, a malicious insider with legitimate publish rights, a compromised registry or, quietly, a pull-through mirror. Mirrors deserve special attention because the verifier and the kubelet frequently do not read from the same place. The verifier may reach the upstream registry with one set of credentials while nodes are configured to pull through a cache. Same tag, two resolutions, no contradiction anywhere in the logs. ## What the implementations do about it **Kyverno.** `verifyImages` can mutate the image reference to the digest it verified, controlled by `mutateDigest`, and can insist the incoming reference already be a digest via `verifyDigest`. Because it mutates, the rule runs as a mutating admission step, and the admitted object carries the digest. **Sigstore policy-controller.** Its webhook can likewise resolve tags to digests for the resources it admits, so the running spec pins what was checked. **Client-side verification.** Verifying with a CLI in a pipeline is fine, and it closes nothing in the cluster by itself. Its output must feed forward: the pipeline records the digest it verified and deploys *that*, rather than deploying a tag and hoping the tag still means the same thing. ## The second-order problems Mutation solves the naming, and introduces its own ordering questions. - **Other mutating webhooks.** Anything that also rewrites the pod's containers — a sidecar injector, an image-rewriting webhook that repoints images at an internal registry — can act after the verifying webhook. Ordering between mutating webhooks is not a guarantee you should build a security control on; reinvocation exists precisely because a later mutation can invalidate an earlier decision. If an image-rewriting webhook runs after verification, the object that gets persisted may name an image nobody verified. - **Re-pulls over the lifetime of the workload.** A pod outlives its admission check. It is rescheduled, the node is replaced, the image cache is evicted. With a digest in the spec, every one of those re-pulls is checked against the same content. With a tag, each one is a fresh, unverified resolution. - **Image caches.** With a tag and a cache-friendly pull policy, a node may run a locally cached layer set that nobody ever checked. With a digest, the identity of the cached image is unambiguous. - **Non-pod paths.** Objects created by controllers, jobs spawned later, and images named in fields your policy's matcher does not cover are all ways an unverified image gets in behind a rule that looks complete. ## What to actually ask for In interviews the strongest answer states the invariant rather than the setting: **what executes must be named by content, not by a name someone else can repoint.** Two ways to get there — require digest-only references and reject tags outright, or let the verifier substitute the digest it verified — and they compose well. Requiring digests is cleaner and pushes work onto the teams writing manifests, which is why most estates start with substitution and tighten later. Then check the seams: which webhooks run after yours, which registries your nodes actually read from, and whether a re-pull six weeks from now still lands on the bytes somebody vouched for.
- The verifier mutates the reference to a digest. Can anything still change the image before it is persisted?Yes. Other mutating webhooks — sidecar injectors, registry-rewriting admission components — can act on the same object, and ordering between mutating webhooks is not something to rely on for a security decision. If one repoints images after your verification step, the persisted object names an image nobody checked. Audit which mutating webhooks touch containers and where yours sits.
- Why does a pull-through mirror make this worse rather than better?Because the verifier and the nodes often read from different endpoints. The verifier may resolve a tag upstream while nodes resolve the same tag through a cache that holds an older or substituted artifact. Nothing looks inconsistent from either side. Pinning to a digest makes the mismatch impossible to hide: the mirror either has those bytes or the pull fails.
- A pod admitted six weeks ago is rescheduled onto a new node. What is verified at that point?Nothing new — admission already happened, and rescheduling does not re-run your policy in any meaningful sense for that workload's image choice. If the spec carries a digest, the re-pull is content-checked against exactly what was verified. If it carries a tag, the new node performs a fresh resolution that no verifier ever saw.
Checking a parcel's contents and then telling the courier to deliver whatever is in locker 12 today. You inspected a parcel; you did not commit to the one that gets delivered.
saying these in an interview costs you the question
- Believes verifying a tag implicitly pins the digest
- Assumes the node pulls from the same endpoint the verifier read
- Relies on mutating webhook ordering as a security guarantee
- Thinks admission-time verification covers later re-pulls of a tag
- Confuses signing the tag with pinning the content