skip to content

In Kubernetes, how does an imagePullSecrets reference reach the kubelet, and why does a pull secret that works in one namespace fail in another?

level: middleimportance: must knowfreq 62%

answer

  1. LocalObjectReference, no namespace field
  2. dockerconfigjson type and key
  3. ServiceAccount copied at admission only
  4. no merge when pod lists its own
  5. missing Secret still attempts the pull

basics

~20 s

A pod's imagePullSecrets names Secrets of type kubernetes.io/dockerconfigjson in its own namespace. The kubelet reads them and passes matching registry credentials to the runtime. Because the reference is namespace-local, a Secret in another namespace is never found.

solid answer

~50 s

`spec.imagePullSecrets` is a list of Secret names, with no namespace field, so each one is looked up in the pod's own namespace. The Secret must be of type `kubernetes.io/dockerconfigjson` and hold a `.dockerconfigjson` key whose registry host matches the image's host. You can set the list on the pod, or on its ServiceAccount's `imagePullSecrets`. In the ServiceAccount case, the ServiceAccount admission plugin copies the list into the pod at creation time, and only if the pod lists none of its own. The two lists are never merged, and editing the ServiceAccount later does nothing to existing pods. If a named Secret is missing, the kubelet emits a `FailedToRetrieveImagePullSecret` Warning and still tries the pull without credentials. That attempt then fails with an authorization error. So when a pull works in `team-a` but not `payments-authz`, the usual cause is that the Secret was never copied into `payments-authz`, or that the pod runs under a ServiceAccount that does not list it.

code

bash · 5 lines
bash
kubectl create secret docker-registry authz-registry-pull \
  --from-file=.dockerconfigjson=./registry-config.json -n payments-authz
kubectl patch serviceaccount payments-authz -n payments-authz \
  -p '{"imagePullSecrets":[{"name":"authz-registry-pull"}]}'
kubectl rollout restart deployment/payments-authz -n payments-authz

go deeper

for a junior

Recall that the pull secret is a dockerconfigjson Secret, that the pod names it in imagePullSecrets, and that it must live in the pod's own namespace.

for a middle

Explain the ServiceAccount admission copy: it happens only at creation, only when the pod's list is empty, with no merge. Say what the kubelet does when a named Secret is missing.

for a senior

Show a diagnosis path: read the injected list from the live pod, check the Secret's type and host key, spot a node pool that differs in credential plugins, and recreate pods after the fix.

for a principal

Weigh copying Secrets into every namespace against node-level credential plugins, trading tenant isolation against operational load across many teams.

## The pieces involved A pull from a private registry needs three Kubernetes objects to line up, plus the node: - A **Secret** of type **`kubernetes.io/dockerconfigjson`**. Its data key **`.dockerconfigjson`** holds a JSON document in the same format as a client's `config.json`: a map from registry host to credentials. - A **pod spec** whose **`spec.imagePullSecrets`** lists that Secret by name. - Optionally, a **ServiceAccount** whose own **`imagePullSecrets`** field lists it, so that pods don't have to. - The **kubelet** on the node the pod lands on. It fetches the Secrets, picks the credentials whose host matches the image reference, and hands them to the container runtime with the pull request. How the registry checks those credentials is the registry's business and belongs to the private-registry topic. This question covers only how the credentials get from the API to the pull. ## Why it is namespace-local `imagePullSecrets` entries are **`LocalObjectReference`s**: a name and nothing else. A pod can only reference Secrets in its own namespace, and there is no syntax to reach across. That is deliberate. Namespaces are the tenancy boundary, and letting a pod name another team's Secret would leak credentials between tenants. So on a 210-node multi-tenant cluster, creating `authz-registry-pull` in `platform-shared` does nothing for the payments-authorization Deployment in `payments-authz`. Every namespace that pulls needs its own copy of the Secret. Teams usually automate that with a sync controller, or with node-level credentials (below). ## The ServiceAccount path When the **ServiceAccount admission plugin** processes a new pod, it follows this order: 1. It resolves the pod's `serviceAccountName`, or `default` if none is set. 2. If the pod's own `imagePullSecrets` list is **empty**, it copies the ServiceAccount's `imagePullSecrets` into the pod. 3. If the pod already lists any pull secret, it leaves the list alone. There is **no merge**. This step runs **once, at creation**. The consequences: - Adding a Secret to the `default` ServiceAccount does not fix the seven pods already stuck. You must recreate them, for example with `kubectl rollout restart`. - A Deployment that switches to a dedicated ServiceAccount loses the pull secrets that `default` used to inject. - A chart that sets one `imagePullSecrets` entry on the pod silently hides the ServiceAccount's entries. ## What the kubelet does when something is wrong | Situation | What you see | |---|---| | Named Secret does not exist in the namespace | `FailedToRetrieveImagePullSecret` Warning on the pod; the pull is still attempted without those credentials | | Secret exists but its host key does not match the image host (for example `registry.example.com` vs `registry.example.com:5000`) | No credentials are sent; the registry rejects an anonymous pull (401/403 or a not-found style denial) | | Credentials are present but expired or lack pull rights | `Failed` event with an authorization error, then `ImagePullBackOff` | | Secret is the wrong type (for example `Opaque` with a `config.json` key) | Not used as registry credentials; same outcome as missing credentials | The kubelet can read these Secrets only because the **Node authorizer** lets a kubelet fetch Secrets referenced by pods bound to its own node. You do not grant that access yourself. ## Node-level alternatives Per-namespace Secrets are not the only source of credentials. The kubelet also supports **credential provider plugins**, configured with `--image-credential-provider-config`, which fetch short-lived registry tokens for images that match configured patterns. Those credentials apply to every pod on the node, and that is exactly why they suit a platform-wide internal registry and are a poor fit for per-tenant access control. A node pool with the plugin and a node pool without it is a classic cause of "works on some nodes". ## Diagnosis checklist 1. Run `kubectl describe pod` and look for `FailedToRetrieveImagePullSecret` or an authorization error in the `Failed` event. 2. Run `kubectl get pod -o jsonpath='{.spec.imagePullSecrets}'` to see what admission actually injected. 3. Run `kubectl get secret <name> -n <ns>` to confirm the Secret exists and its type is `kubernetes.io/dockerconfigjson`. 4. Decode `.dockerconfigjson` and check that the host key matches the image's registry host exactly. 5. After fixing, recreate the pods. Running pods never re-read their ServiceAccount.

  • You added the pull secret to the ServiceAccount, but the seven stuck pods still fail. Why?
    The ServiceAccount admission plugin copies `imagePullSecrets` into a pod only when the pod is created. The existing pods were admitted with an empty or different list, and a pod's `imagePullSecrets` cannot be changed on a live pod. Recreate them, typically with `kubectl rollout restart deployment/...`, and check the new pods' `spec.imagePullSecrets`.
  • The pod spec lists one pull secret and the ServiceAccount lists another. Which credentials does the kubelet get?
    Only the pod's own. The admission plugin fills `imagePullSecrets` from the ServiceAccount only when the pod's list is empty, so a non-empty pod list completely hides the ServiceAccount's entries. If both are needed, list both on the pod, or drop the pod-level entry and rely on the ServiceAccount.
  • When would you use a kubelet credential provider plugin instead of per-namespace pull Secrets?
    When one internal registry should be pullable by every workload on the node and you want short-lived tokens instead of long-lived Secrets copied into every namespace. The kubelet calls the plugin for images matching configured patterns. The tradeoff is scope: the credentials apply to every pod on those nodes, so per-tenant separation needs Secrets or registry-side rules instead.

saying these in an interview costs you the question

  • A pod can reference a pull Secret from another namespace by name
  • An Opaque Secret holding config.json works as pull credentials
  • ServiceAccount pull secrets are merged with the pod's own list
  • Editing the ServiceAccount fixes pods that are already running
  • A missing pull Secret makes the kubelet skip the pull entirely
  • The workload needs RBAC to read its pull Secrets