An application container in a Kubernetes Pod sends all of its outbound traffic through a proxy container in the same Pod, and it fails its first few requests every time the Pod starts. Explain why, and how you make the proxy fully up before the application container starts.
answer
- containers[] start in parallel — no ordering
- Native sidecar = initContainers + restartPolicy: Always
- startupProbe turns 'started' into 'serving'
- Retry with backoff still needed for mid-life restarts
- sleep in entrypoint = tuned race, not a fix
basics
~20 sContainers listed in a Pod's containers field start in parallel, so the app can beat the proxy. Fix it by declaring the proxy in initContainers with restartPolicy: Always and a startupProbe — the kubelet then waits for the probe to pass before starting the app.
solid answer
~50 sThe kubelet starts every entry in `containers` **concurrently**; list order means nothing. So the app process can open its first connection before the proxy has bound its listener, and those requests fail — a startup race, not a code bug. The correct fix is a **native sidecar**: move the proxy into `initContainers` with `restartPolicy: Always`. The kubelet then walks the init list in order and does not start the app container until the proxy has started. Crucially, add a `startupProbe` to the proxy — without one the kubelet waits only for the container to be *started*, not for it to be *serving*, and a proxy that must fetch config from a control plane is not serving at process start. Supplementary measures: retry with backoff in the client (you want that anyway for mid-life proxy restarts), and a `readinessProbe` on the proxy so Pod readiness reflects it. On clusters that predate native sidecars you fall back to app-side retry or a blocking wrapper script.
code
yaml · 20 linesspec:
initContainers:
- name: proxy
image: registry.example.com/proxy:1.21
restartPolicy: Always
ports:
- containerPort: 15001
startupProbe:
httpGet: { path: /healthz/ready, port: 15021 }
periodSeconds: 1
failureThreshold: 60 # up to 60s for config fetch
readinessProbe:
httpGet: { path: /healthz/ready, port: 15021 }
periodSeconds: 10
containers:
- name: app
image: registry.example.com/app:4.2.0
env:
- name: HTTP_PROXY
value: http://127.0.0.1:15001go deeper
State the core fact — containers in a Pod start in parallel, so the app can outrun the proxy — and that the platform fix is a native sidecar.
Add the mechanics: initContainers + restartPolicy: Always, and that a startupProbe is what makes the wait mean "serving" rather than "launched".
Diagnose from evidence (startedAt timestamps, correlated logs), cover the shutdown mirror image, and argue that client retry remains necessary for mid-life sidecar restarts.
Decide where the concern belongs at all — per-Pod sidecar vs node-level agent vs library — and set a platform-wide standard for probe budgets, grace periods and the minimum cluster version that standard assumes.
## Why the race exists A Pod is a group of containers sharing a network namespace. The kubelet's contract for the `containers` list is deliberately weak: it creates and starts all of them, and the order in which their processes reach a usable state is undefined. There is no `dependsOn`. So in a Pod with `[app, proxy]`, the app's first outbound connection may arrive before the proxy has parsed its config, fetched routes from a control plane, and bound `127.0.0.1:15001`. The connection is refused, or — with transparent iptables/eBPF redirection installed — is redirected to a listener that isn't there yet. The symptom is characteristic: failures cluster in the first one to ten seconds of a Pod's life, are absent afterwards, get worse on cold nodes and under image-pull pressure, and vanish when you add a `sleep` to the app's entrypoint (which is the diagnosis, not the fix). ## Ordered startup, the supported way Move the proxy from `containers` into `initContainers` and set `restartPolicy: Always` on it. That makes it a **native sidecar**: it is started in init-list order, it keeps running for the Pod's lifetime (unlike a normal init container, which must exit), and the kubelet does not proceed to the next init entry or to the app containers until it has started. The subtlety is what "started" means. By default it means the container process is running and its `postStart` hook, if any, has returned. That is a weaker condition than "the proxy is serving traffic". To make the gate real, declare a `startupProbe` on the sidecar pointing at its own health endpoint; the kubelet then blocks until that probe succeeds. Choose `failureThreshold × periodSeconds` to comfortably exceed worst-case config-fetch time, because exceeding it kills and restarts the sidecar and you loop. Ordering composes: if the proxy itself needs a certificate that another container fetches, declare `[cert-fetcher (plain init), proxy (sidecar), app]` and each step is gated on the previous one. ## Why you still want client-side retry Ordered startup fixes *Pod start*. It does not fix a proxy that crashes and restarts at hour six, a control-plane blip that makes the proxy briefly reject routes, or a rolling upgrade of the sidecar image. Any client that treats "connection refused to my own sidecar" as fatal will still page someone. Bounded retry with exponential backoff and jitter on connect-level errors is the durable half of the answer; ordered startup is the half that removes the guaranteed failure at t=0. Add a `readinessProbe` to the sidecar too. Sidecar readiness participates in Pod readiness, so a Pod whose proxy is unhealthy is removed from Service endpoints instead of blackholing traffic. ## The workarounds you should be able to name — and criticise On clusters without native sidecars (before 1.29 in practice), teams used: - **A blocking init container** that polls the proxy's health endpoint. This mostly does not work for an in-Pod proxy, because the proxy lives in `containers` and therefore hasn't been started yet while init containers run — it deadlocks. It only works when the dependency is *outside* the Pod. - **A wrapper entrypoint** in the app image that curls `localhost:15021/healthz/ready` in a loop before exec'ing the real process. Effective, but it puts sidecar knowledge into every application image. - **Mesh-specific flags** such as Istio's `holdApplicationUntilProxyStarts`, which injects a `postStart` hook on the proxy that blocks until it is ready. Works, but it is a vendor-specific patch for a missing platform primitive. - **A fixed `sleep`** in the entrypoint. Always wrong: it is a race you tuned rather than removed, and it slows every rollout. ## Shutdown is the mirror image The same Pod has a symmetric bug at the other end: with plain `containers`, all containers get SIGTERM together, so the proxy can die while the app is still draining, and the last requests fail. Native sidecars fix this too — regular containers are terminated first, then sidecars in reverse declaration order. If you are re-declaring the proxy to fix startup ordering, you get the drain ordering for free; say so in an interview, because a candidate who only fixes half the race has only half the model. ## What to check when diagnosing `kubectl describe pod` gives you each container's start time and restart count. Application logs with timestamps against the proxy's own "listener ready" log line prove the ordering. `kubectl get pod -o jsonpath` over `.status.containerStatuses[*].state.running.startedAt` shows the actual concurrency. And a single `kubectl exec` into the app container calling the proxy's admin port confirms whether the dependency is reachable once things settle.
- Why doesn't a plain init container that polls the proxy's health endpoint solve this?Because the proxy is inside the same Pod, in the `containers` list, and nothing in `containers` is started until every init container has finished. The poll would wait for a listener that by definition cannot exist yet, so the Pod deadlocks until the init container's own timeout. Polling from an init container only works for dependencies outside the Pod.
- You added the proxy as a native sidecar but the app still fails one request in a thousand at startup. What now?Without a `startupProbe`, the kubelet gates only on the container having started, which for a proxy that fetches its configuration remotely is earlier than "serving". Add a `startupProbe` against the proxy's ready endpoint so the gate reflects actual readiness, and keep bounded retry in the client for the residual cases — sidecar restarts and control-plane hiccups mid-life.
It is like a shop and its delivery courier opening at the same time: the shop can hand over a parcel before the courier's van has arrived. You want the courier's van confirmed on the forecourt before the shop unlocks its doors.
saying these in an interview costs you the question
- Claiming the order of entries in the `containers` list determines startup order.
- Adding a fixed `sleep` to the app entrypoint and calling it fixed.
- Using a plain init container to poll a proxy that lives in the same Pod's `containers` list — that deadlocks.
- Assuming a native sidecar without a startupProbe guarantees the proxy is serving before the app starts.
- Fixing startup ordering but ignoring the symmetric shutdown race that drops in-flight requests.