What does a Kubernetes container's imagePullPolicy control, what is its default, and how does it interact with images already cached on the node?
answer
- three values, one node-local cache
- default written at admission, not runtime
- latest or no tag and no digest
- re-pushed tag, warm vs cold nodes
- Always resolves, reuses cached layers
basics
~20 simagePullPolicy tells the kubelet when to check the registry: Always on every start, IfNotPresent only when the image is not cached, Never not at all. Unset, it defaults to Always for :latest or images with no tag and no digest, otherwise IfNotPresent.
solid answer
~50 s`imagePullPolicy` is a per-container field that tells the kubelet when to ask the container runtime to pull. `Always` resolves the image reference against the registry on every container start, but the runtime fetches only the layers the node is missing. `IfNotPresent` pulls only when the node has no copy of the image. `Never` never pulls, and fails with `ErrImageNeverPull` if the image is missing. If you leave the field out, the API server fills it in when the pod is created. It becomes `Always` when the tag is `:latest`, or when the image has neither a tag nor a digest. Anything else gets `IfNotPresent`, and that includes a digest-only reference. The trap is the node cache. With `IfNotPresent`, a tag that was pushed again means nodes that already hold it keep running the old code. It also means a registry outage only breaks nodes that do not have the image yet.
code
bash · 2 lineskubectl get pods -n payments-authz -l app=payments-authz \
-o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,POLICY:.spec.containers[0].imagePullPolicy,IMAGEID:.status.containerStatuses[0].imageID'go deeper
Recall the three values and the default rule: Always for latest or an image with no tag and no digest, IfNotPresent for everything else. Say that the cache belongs to the node.
Explain that the API server writes the default at creation, and that Always resolves the manifest but reuses cached layers. Walk through what a re-pushed tag does under IfNotPresent.
Show that you would diagnose a split-version or node-dependent failure by comparing imageID across replicas, and that you understand Always turns the registry into a dependency for every restart.
Argue for digest pinning or immutable tags as platform policy, and weigh the cross-tenant exposure of a shared node cache against the credential re-check the kubelet now performs.
## What the field is Every container in a Kubernetes pod spec has an **`imagePullPolicy`** field. Before the **kubelet** (the node agent) starts that container, it reads the field to decide whether to ask the **container runtime** (containerd or CRI-O, behind the CRI) to pull the image from its registry. The alternative is to use a copy already in the node's local **image cache**, meaning the images the runtime has stored on disk. | Value | What the kubelet does | Failure when the registry is unreachable | |---|---|---| | `Always` | Asks the runtime to resolve the reference on every container start; layers already on disk are not downloaded again | The container cannot start, even on a node that has the image cached | | `IfNotPresent` | Pulls only if the node does not already have the image | Only nodes without the image fail | | `Never` | Never pulls; uses the local copy only | Registry is irrelevant; a missing image gives `ErrImageNeverPull` | ## How the default is chosen The default is not applied by the kubelet at start time. The API server writes it into the stored pod spec when the object is created, so `kubectl get pod -o yaml` always shows a concrete value. The rule is: 1. If the image's tag is `latest`, the policy becomes **`Always`**. 2. If the image has **no tag and no digest**, it is treated as `:latest`, so it also becomes **`Always`**. 3. Every other case becomes **`IfNotPresent`**. That covers an explicit tag such as `:2.14.3` and a reference pinned only by digest (`@sha256:...`). The rule exists because `latest` is a moving tag: if you ask for it, you probably want whatever it points to now. A specific tag or digest is assumed to be stable, so the cached copy is trusted. ## Where the node cache bites Take a payments-authorization API run as a **7-replica Deployment** on a 210-node cluster. The image tag is `authz:2.14.3-rc`, so the policy defaults to `IfNotPresent`. - **A re-pushed tag.** Someone fixes a bug and pushes `2.14.3-rc` again. Pods scheduled onto nodes that already hold the old image start the **old** code, and pods on fresh nodes start the **new** code. All seven replicas show the same tag in their spec, so nothing in `kubectl get pods` shows the split. Only the image ID in `status.containerStatuses[].imageID` differs. - **A registry outage hides on warm nodes.** Replicas on nodes that cached the image restart normally. Replicas on nodes without it go to `ErrImagePull` and then `ImagePullBackOff`. The incident looks random until you map the failing pods to nodes. - **`Never` needs pre-loaded images.** It suits air-gapped or pre-baked node images. On any node where nothing loaded the image, the pod fails with `ErrImageNeverPull`, and the event says the image is not present with pull policy of Never. ## Why Always is cheaper than it sounds Many engineers avoid `Always` because they think it downloads the whole image on every restart. It does not. The runtime resolves the tag to a manifest digest and downloads only the layers it lacks, so an unchanged image costs one small registry round trip. The real cost of `Always` is a **dependency**: every container start, including a restart after a crash, needs the registry to be reachable and the credentials to be valid. ## Private images and the shared cache The image cache belongs to the node, not to a namespace. Historically, `IfNotPresent` let a pod that had **no** pull credentials run a private image that another tenant's pod had already pulled onto the same node. The kubelet feature gate **`KubeletEnsureSecretPulledImages`** closes that gap. It was Alpha in 1.33 and has been Beta, enabled by default, since 1.35. With it, the kubelet records which credentials pulled each image and checks that a new pod is entitled to it before reusing the cached copy. The kubelet config field **`imagePullCredentialsVerificationPolicy`** controls how strict that check is. Its default, `NeverVerifyPreloadedImages`, exempts only images that something other than the kubelet put on the node. ## Practical guidance - Pin production images by **digest**, or use tags your registry makes immutable. Then `IfNotPresent` is both safe and fast. - Never ship `:latest` to production. It forces `Always`, and it makes "which code is running" impossible to answer from the spec. - If you need `Always` for mutable tags, accept that a registry outage now blocks restarts, and plan for it. - When pods behave differently on different nodes, compare `imageID` across replicas before anything else.
- You changed a pod's image from a tag to a digest with imagePullPolicy unset. Which policy does it get, and is that safe?It gets `IfNotPresent`, because the API server applies `Always` only when the tag is `latest` or when there is neither a tag nor a digest. That is safe: a digest names exactly one manifest, so a cached copy with that digest is by definition the right content. Digest pinning is the cleanest way to make `IfNotPresent` correct, because a re-push cannot change what the reference means.
- Why might a pod with no pull secret run a private image on one node but fail on another?The image cache belongs to the node. With `IfNotPresent`, a node where another pod already pulled the private image could let this pod reuse it without any credentials, while a cold node has to pull and gets refused. Current kubelets narrow this with the `KubeletEnsureSecretPulledImages` gate (on by default since 1.35), which checks that the new pod is entitled to the cached image before reusing it.
IfNotPresent is like cooking from what is already in your fridge and only shopping for what is missing: fast, but if the recipe changed since you last shopped, you still cook the old version.
saying these in an interview costs you the question
- The default imagePullPolicy is always IfNotPresent, regardless of the tag
- Always downloads the full image again on every container restart
- A tag uniquely identifies image content, so re-pushing a tag is harmless
- The kubelet decides the default at start time; the stored spec stays empty
- Never pulls the image once and then caches it for later
- Pinning by digest still forces Always because the digest must be checked