skip to content

An Istio-injected service logs connection failures on its first few outbound calls immediately after each pod starts, then behaves normally. What ordering problem causes this, and what does Istio offer to fix it?

level: seniorimportance: should knowfreq 48%

answer

  1. failures only in the first seconds
  2. capture is ready before the proxy is
  3. app and sidecar start concurrently
  4. make the proxy start first, exit last
  5. Jobs that never finish are the same bug

basics

~20 s

The application container starts before istio-proxy is ready, so early outbound calls hit redirection rules pointing at a proxy that cannot yet serve them. Fix it with holdApplicationUntilProxyStarts, or by running istio-proxy as a Kubernetes native sidecar container, which Kubernetes starts first.

solid answer

~50 s

Traffic capture is installed before any container runs, but the proxy that capture points at needs a second or two to bootstrap and receive its configuration from istiod. Kubernetes starts the app container in parallel, so a service that calls a dependency at startup — a database, a config service, a discovery registry — hits redirected connections that the proxy cannot serve yet, and fails until Envoy is up. Istio's direct fix is `holdApplicationUntilProxyStarts`, which makes the proxy block the pod's startup sequence until Envoy is listening; set it mesh-wide in mesh config or per pod through the `proxy.istio.io/config` annotation. The structural fix, where your Kubernetes and Istio versions support it, is running istio-proxy as a **native sidecar container** — an init container with `restartPolicy: Always` — so Kubernetes guarantees it starts before the app and stops after it. The mirror-image bug is shutdown: a Job whose sidecar keeps running never completes.

code

yaml · 19 lines
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
      annotations:
        # Block the app container until Envoy is listening
        proxy.istio.io/config: '{"holdApplicationUntilProxyStarts": true}'
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2.9.0

go deeper

for a junior

Recognise that a pod has more than one container and that they do not become useful at the same instant. Know that Istio has a setting to make the app wait for its proxy.

for a middle

Explain that capture rules exist before Envoy is ready, name holdApplicationUntilProxyStarts and where it is set, and describe how you would confirm the diagnosis from the two containers' logs.

for a senior

Show the full lifecycle picture: startup ordering, native sidecar containers, drain behaviour on shutdown, and the Job-never-completes variant. Diagnose from timestamps rather than guessing.

for a principal

Decide the mesh-wide default and own its consequences — added startup latency across every scale-out event versus deterministic ordering — and set the standard for how batch workloads are meshed at all.

## Why the race exists at all The redirection rules are in place before any container starts — that part is reliable. What is not reliable is that the thing they redirect *to* is ready. `istio-proxy` must start Envoy, fetch its workload certificate, and receive its initial configuration from `istiod` before it can proxy anything. Classic Kubernetes semantics start all of a pod's regular containers concurrently, so your application may be issuing its first outbound call while Envoy is still bootstrapping. The connection is captured and fails. The symptom is unmistakable once you have seen it: failures clustered in the first seconds of a pod's life, on outbound calls only, disappearing on their own, and never reproducible under load. Applications that connect to a database, warm a cache, or register with a discovery service during initialisation are the ones that get hurt; applications that only touch the network when the first request arrives usually never notice, because by then the proxy is up. ## The direct fix `holdApplicationUntilProxyStarts` makes the injected proxy delay the rest of the pod's startup until Envoy is listening. You can set it mesh-wide in the mesh configuration's proxy defaults, or per workload: ```yaml template: metadata: annotations: proxy.istio.io/config: '{"holdApplicationUntilProxyStarts": true}' ``` It is effective and it is what most teams reach for. The cost is a couple of seconds added to every pod's startup, which matters when you are scaling out under load or draining a node — a fleet-wide rollout of a slow-starting service gets slower still. ## The structural fix Kubernetes eventually grew a first-class answer: **native sidecar containers**, expressed as an init container with `restartPolicy: Always`. The kubelet starts such a container before the regular containers, keeps it running for the pod's lifetime, and terminates it only after the regular containers have exited. That is exactly the lifecycle a service-mesh proxy wants, and Istio can run `istio-proxy` in that shape where the feature is available and enabled on both sides. It fixes startup ordering by construction rather than by a cooperative wait — and it fixes shutdown at the same time. Check versions before promising this in an interview: the Kubernetes feature moved through alpha, then beta-on-by-default, then general availability across successive releases, and Istio's use of it is gated on the control plane as well. State the assumption rather than asserting a default. ## The mirror-image bug: shutdown The same lifecycle gap runs in the other direction and is often the more painful of the two: - **Jobs and CronJobs never complete.** The application container exits successfully, the sidecar keeps running, the pod stays `Running` forever, and the Job never reports completion. Under classic sidecar semantics the workaround is for the job to ask the agent to exit — a POST to `/quitquitquit` on the proxy's agent port 15020 as the last step of the script. Native sidecars remove the problem, because Kubernetes stops them once the regular containers have finished. - **In-flight requests are cut short.** If the proxy stops before the app finishes draining, the last requests fail during a rolling update. Istio addresses this with a drain period during which the proxy keeps serving existing connections after receiving the termination signal, which must be shorter than the pod's termination grace period or the kubelet kills everything anyway. ## How to diagnose it in practice 1. Correlate failure timestamps against pod start time — if every failure is within the first seconds, ordering is your prime suspect. 2. Compare the app container's log with the `istio-proxy` container's log for the same pod; the proxy prints when Envoy is ready, and the app's failures should stop right around that line. 3. Confirm it is outbound-only. Inbound requests are usually unaffected because nothing routes to the pod until its readiness gate passes. 4. Test the hypothesis cheaply by adding the hold annotation to one Deployment and restarting it. ## The judgment interviewers are listening for A retry loop around startup calls is not wrong — a service that cannot survive a dependency being briefly unavailable is fragile regardless of the mesh — but it is not the answer to this question. The interviewer wants to hear that a transparent proxy inserts a *lifecycle* dependency, not just a network hop: something in the pod must now be alive before your code's first packet and must outlive your code's last one. Everything else — the annotation, native sidecars, `/quitquitquit`, drain timing — follows from that single observation.

  • What is the cost of turning holdApplicationUntilProxyStarts on for the whole mesh?
    Every pod's startup gains the proxy's bootstrap time — typically a couple of seconds. That is invisible for steady-state workloads but compounds when you scale out under load, drain a node, or roll a large Deployment, since the delay is paid per pod. Many teams enable it mesh-wide anyway and treat the seconds as the price of deterministic startup.
  • Why does a Kubernetes Job with an injected sidecar never complete?
    The job's container exits, but istio-proxy keeps running, so the pod never reaches a terminal phase and the Job is never marked complete. Under classic sidecar semantics the script must tell the agent to exit — a POST to /quitquitquit on port 15020 as its final step. Running the proxy as a native sidecar container removes the problem, because Kubernetes stops it once the regular containers finish.
  • Isn't a retry loop in the application the real fix?
    Retries are good hygiene and you should have them, but they treat the symptom. The mesh has introduced a lifecycle dependency — something must be alive before your first packet and after your last — and a retry loop neither expresses nor guarantees that. It also does nothing for the shutdown half of the problem, where retries cannot help at all.

saying these in an interview costs you the question

  • Blames the dependency being slow rather than the proxy's startup
  • Assumes traffic capture readiness implies the proxy is ready
  • Thinks a readiness probe on the app prevents outbound startup failures
  • Says retries alone solve it, ignoring the shutdown half
  • Believes Kubernetes always starts sidecars before app containers

context

resolving links…