A newly created pod in a namespace you believe is part of the Istio mesh came up with no istio-proxy container. Explain how Istio decides which pods get a sidecar, and how you would find the reason this one did not.
answer
- decided at admission, never afterwards
- namespace label brings the pod into scope
- pod label is the opt-out
- existing pods are never retro-injected
- revision label after a canary upgrade
basics
~20 sIstio injects through a mutating admission webhook selected by namespace labels — istio-injection=enabled or istio.io/rev=<revision> — with the pod-level sidecar.istio.io/inject label as an override. Check those labels, whether the pod predates them, revision mismatch after an upgrade, host-network pods, and webhook availability.
solid answer
~50 sInjection is an admission-time decision. Istio's mutating webhook is scoped by selectors, so the namespace label is what brings a pod into consideration: `istio-injection=enabled` for the default control plane, or `istio.io/rev=<revision>` when you run revisioned or canary control planes. Within a selected namespace, a pod-level `sidecar.istio.io/inject: "false"` label opts that workload out. To diagnose, I check the namespace's labels, then the pod's own labels and annotations, then whether the pod was created *after* labelling — existing pods are never retro-injected, so a `kubectl rollout restart` is often the whole fix. After a canary upgrade the usual culprit is a namespace still labelled for a revision that no longer exists. I also check host-network pods, which are skipped, and whether the webhook itself was reachable: with a permissive failure policy an unavailable istiod produces silently uninjected pods. `istioctl analyze` catches most of these in one command.
code
yaml · 18 linesapiVersion: apps/v1
kind: Deployment
metadata:
name: batch-reporter
namespace: payments # namespace labelled istio.io/rev=stable
spec:
selector:
matchLabels:
app: batch-reporter
template:
metadata:
labels:
app: batch-reporter
sidecar.istio.io/inject: "false" # opt this workload out of the meshed namespace
spec:
containers:
- name: reporter
image: registry.example.com/batch-reporter:1.4.2go deeper
Know that a namespace label turns injection on and that pods must be recreated afterwards. Be able to check namespace and pod labels with kubectl and say what you are looking for.
Explain the webhook mechanism and the label precedence: namespace label selects, pod-level sidecar.istio.io/inject overrides. Walk the diagnosis in order and name revision labels as an upgrade-time cause.
Show that you treat a silently uninjected pod as a security event, not a cosmetic one. Discuss webhook failure policy as a real availability-versus-enforcement choice and how you detect unmeshed workloads.
Own the policy: which namespaces are meshed by default, how revision labels are managed across a canary control-plane upgrade, and what guardrail stops a team from shipping unmeshed workloads into a mesh that is relied on for identity.
## Injection is an admission-time decision Istio does not scan running pods. It registers a **mutating admission webhook** with the Kubernetes API server, and the API server calls it while a pod object is being created. If the call happens and the policy says yes, the stored pod spec comes back with `istio-proxy` added. If the call never happens, the pod is stored exactly as written and stays unmeshed forever. Every diagnosis follows from that one fact. ## What selects a pod **The namespace label is the gate.** For a default installation the label is `istio-injection=enabled`. For a revisioned installation — the normal shape once you have done a canary control-plane upgrade — the label is `istio.io/rev=<revision>`, for example `istio.io/rev=1-24-1` or an alias such as `istio.io/rev=stable`. The two schemes are separate: a namespace carrying the legacy `istio-injection=enabled` label is claimed by the default-revision webhook, and a revisioned control plane will not inject into it. **The pod label is the override.** `sidecar.istio.io/inject: "false"` on the pod template excludes that workload from a meshed namespace — the standard way to exempt a batch job or an agent that cannot tolerate the proxy. (The same key also exists in the older annotation form.) Opting a single pod *in* from an unlabelled namespace is the direction that usually does not work, because the webhook is not consulted for that pod at all. **Some pods are skipped by design.** Pods on the host network are not injected: their network namespace is the node's, so capture rules would rewrite node traffic. ## The diagnostic order that finds it fastest ```bash kubectl get ns payments --show-labels # istio-injection / istio.io/rev present? kubectl get pod -n payments checkout-abc -o yaml # sidecar.istio.io/inject on the pod? kubectl get mutatingwebhookconfiguration # does the injector webhook exist? istioctl analyze -n payments # flags most of the above in one pass ``` 1. **Namespace labels.** The single most common cause is simply a missing or misspelled label, or the label applied to the wrong namespace. 2. **Pod predates the label.** Labelling a namespace changes nothing about pods that already exist. A Deployment must produce new pods — `kubectl rollout restart deployment/checkout` — before anything is injected. Teams routinely label a namespace, see no sidecars, and conclude injection is broken. 3. **Revision mismatch.** After a canary upgrade, the old revision is uninstalled while namespaces still point at it. Their pods then get nothing, because no running webhook claims that revision. This is the failure that bites during upgrades, and it is silent: pods are `Running` and `1/1`, they just are not in the mesh. 4. **Pod-level opt-out.** Someone added `sidecar.istio.io/inject: "false"` to the pod template months ago to work around a problem nobody remembers. 5. **Webhook availability and scope.** If the webhook's failure policy is permissive and istiod was unavailable when the pod was admitted, the API server proceeds without injection. Result: an unmeshed pod created during a control-plane outage, sitting next to correctly meshed siblings from an hour earlier. A stricter failure policy converts that into a visible pod-creation error instead — safer for a mesh you rely on for mTLS, riskier for cluster availability. ## Why the offline path exists `istioctl kube-inject` renders the injected spec locally, so you can apply an already-injected manifest. It is useful when webhooks are unavailable or when a pipeline wants the exact spec under review, but it freezes the sidecar configuration at render time: pods keep whatever proxy image and settings were current when the manifest was produced, and they will not pick up new injection templates. Webhook injection is the default for exactly that reason. ## The operational consequence worth stating An unmeshed pod does not fail — it works, without mTLS, without policy enforcement, and without mesh telemetry. In a mesh where `PeerAuthentication` is permissive, it can talk to meshed peers in plaintext and nobody notices. That is why mature installations alert on the ratio of injected to non-injected pods per namespace rather than trusting that a label was applied, and why the strict failure policy is the defensible default once the mesh carries security guarantees.
- You labelled the namespace and no sidecars appeared. What is the most likely reason?The pods already existed. Injection happens when a pod object is created, so labelling a namespace has no effect on running pods. Restart the workloads — `kubectl rollout restart` on the Deployments — and the replacement pods go through admission and come back injected.
- How does the webhook's failure policy change what you see when istiod is down?With a permissive policy the API server proceeds without calling the injector, so pods created during the outage come up unmeshed and healthy-looking — the worst kind of failure, because it is silent and the pods bypass mTLS. A strict policy fails pod creation instead, which is loud but blocks scheduling while the control plane is unavailable.
- When would you use istioctl kube-inject instead of the webhook?When you need the injected spec materialised ahead of admission — an environment without the webhook, or a pipeline that wants the exact rendered pod under review. The trade-off is that the sidecar image and configuration are frozen at render time, so pods drift from the current injection template until the manifests are regenerated.
saying these in an interview costs you the question
- Thinks labelling a namespace injects sidecars into existing pods
- Believes a pod label alone meshes a pod in an unlabelled namespace
- Ignores revision labels and assumes one global control plane
- Assumes an uninjected pod will visibly fail or crash
- Says injection happens when the container starts, not at admission