Pods in a namespace fail with `ErrImagePull` from a private registry. Explain how Kubernetes authenticates image pulls and how you wire credentials with `imagePullSecrets`.

## Who actually pulls The kubelet on the scheduled node hands the pull to the container runtime (containerd/CRI-O). Neither the API server nor the Pod's own service account is involved in registry authentication. That single fact explains most of the confusion: the credential does not need to be *readable by the app*, it needs to be *findable by the kubelet* through the Pod spec. ## Credential sources, in order The kubelet assembles candidate credentials from: 1. **`imagePullSecrets` on the Pod spec** — a list of Secret names in the Pod's own namespace. 2. **`imagePullSecrets` on the Pod's ServiceAccount** — the ServiceAccount admission controller merges these into the Pod at creation time, so the resulting Pod spec looks identical to case 1. 3. **Node-level credentials** — a credential-provider plugin configured on the kubelet (`--image-credential-provider-config`), which is how EKS/GKE/AKS give nodes access to ECR/Artifact Registry/ACR without any Secret, plus any static config file baked into the node image. The runtime tries the credentials matching the registry host of the image reference. If none matches, it attempts an anonymous pull, which yields a 401/403 surfaced as `ErrImagePull` and then `ImagePullBackOff`. ## The Secret It must be `type: kubernetes.io/dockerconfigjson`, with the payload under the key `.dockerconfigjson` — a Docker config document mapping registry hosts to auths. Build it with `kubectl create secret docker-registry`; hand-writing it goes wrong because the `auth` field is itself base64 of `username:password`, inside a JSON document that is itself base64-encoded in `data`. An `Opaque` Secret containing exactly the same JSON does **not** work — the kubelet selects by type. ## The namespace trap Secrets are namespaced and `imagePullSecrets` can only name a Secret in the Pod's own namespace. There is no cluster-scoped pull secret. So every namespace that runs private images needs its own copy. Teams solve this by: - templating it into each namespace from the platform's namespace-provisioning pipeline; - a replicator controller (e.g. reflector-style tooling) that copies a source Secret into annotated namespaces; - patching the `default` ServiceAccount in each new namespace as part of provisioning; - or eliminating it entirely with node credential providers or workload identity. ## The host-matching trap The registry host in the Secret must match the host in the image reference. `--docker-server=https://index.docker.io/v1/` is the special value for Docker Hub; for anything else it is the bare host (`registry.example.com`, optionally with a port). An image written as `registry.example.com/team/app:1.0` will not match a credential stored under `https://registry.example.com/`. Mismatch here produces the same `ErrImagePull` as a wrong password, which is why the diagnosis is confusing. ## Attaching to the ServiceAccount The scalable form is one patch per namespace: ``` kubectl patch serviceaccount default -n team-a \ -p '{"imagePullSecrets":[{"name":"regcred"}]}' ``` Every Pod created afterwards with that ServiceAccount gets the reference merged in. Two caveats: it applies at Pod-creation time, so existing Pods are unaffected until they are recreated; and it applies per ServiceAccount, so a namespace using several needs each patched. ## Diagnosing a failure Read the events, not the Pod status: `kubectl describe pod` shows the runtime's actual message. `401 Unauthorized` means credentials were sent and rejected — expired token or wrong password. `no basic auth credentials` or an anonymous-pull 403 means nothing matched — wrong namespace, wrong type, missing reference or a host mismatch. `manifest unknown` means auth worked and the tag does not exist. Then verify what the kubelet would use: ``` kubectl get secret regcred -n team-a -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq . kubectl get sa default -n team-a -o jsonpath='{.imagePullSecrets}' ``` ## Two security notes worth raising First, once an image is present in a node's image cache, `imagePullPolicy: IfNotPresent` lets **any** Pod on that node run it without proving it can authenticate — a cross-tenant leak of private images. The `AlwaysPullImages` admission plugin forces a pull (and therefore an authorization check) per Pod; it is the standard hardening for multi-tenant clusters, at the cost of pull traffic. Second, a pull Secret is a registry credential readable by anyone who can read Secrets in that namespace. Prefer short-lived, pull-only credentials, or node credential providers where the node's own cloud identity is exchanged for a short-lived token and no Secret exists at all.