Calls through a Kubernetes Service succeed most of the time, but a small percentage fail with connection resets, and the rate spikes during deployments. What causes this intermittent pattern and how would you fix it?

## Why "a small percentage" is the important clue Total failure has few causes and they are easy to find. Partial failure means the request path is *usually* correct, so the fault is in something that varies per request: which backend was chosen, when the request arrived relative to a state change, or which connection it reused. That narrows the search dramatically. ## Cause 1: the endpoint removal race (the deployment correlation) Deleting a pod triggers several independent sequences: 1. The API server sets `deletionTimestamp`. 2. The kubelet sees it and runs any `preStop` hook, then sends SIGTERM to the container. 3. The EndpointSlice controller sees it and marks the pod's address `ready: false` / `terminating: true`. 4. kube-proxy on **every node** watches EndpointSlices and rewrites iptables or IPVS rules. 5. Any external load balancer or ingress controller updates its own backend list. Steps 2 and 3–5 race. On a small idle cluster the propagation is milliseconds; with thousands of nodes, a busy API server, or a controller under load, it can be seconds. If the application closes its listening socket the instant it receives SIGTERM, every request that kube-proxy sends in that window hits a closed port and the client sees a reset — which is exactly the pattern of a low, deployment-correlated error rate. The standard remedy is counter-intuitive: make the pod keep serving *after* it has been told to die. ```yaml lifecycle: preStop: exec: {command: ["sh", "-c", "sleep 10"]} terminationGracePeriodSeconds: 45 ``` The preStop hook delays SIGTERM, so the pod continues to accept traffic while the not-ready state propagates. The application then handles SIGTERM by refusing *new* connections, finishing in-flight requests, and exiting — and the grace period must exceed preStop plus the longest legitimate request, otherwise the kubelet SIGKILLs mid-request (exit 137 with reason Error) and you have traded one truncation for another. A complementary tactic is flipping readiness to failing *before* deleting the pod (some frameworks expose a shutdown hook that does this), so removal from endpoints begins earlier in the sequence. ## Cause 2: one unhealthy backend kube-proxy spreads new connections across ready endpoints roughly evenly. So one bad replica out of eight gives you about 12.5% failures — a stable, suspiciously round number. "Bad" here means still passing readiness but not actually serving: an exhausted connection pool, a wedged thread pool, a stale config, a half-failed migration, a node with a broken route. Isolate by taking the Service out of the loop: ``` kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{.items[*].endpoints[*].addresses[*]}' # then curl each pod IP directly in a loop and compare ``` If exactly one address fails consistently, you have your pod, and the next question is why readiness did not catch it — usually because the probe is shallower than the failure. ## Cause 3: connection reuse defeats load balancing kube-proxy balances **connections**, not requests. A client with HTTP keep-alive, a connection pool, or HTTP/2 and gRPC (which multiplex everything over a single long-lived connection) picks a backend once and stays there. Consequences: - A newly scaled-up replica receives no traffic from existing clients. - One bad backend affects a fixed subset of clients completely rather than everyone slightly. - During a rollout, clients holding connections to terminating pods experience the reset at the moment of shutdown — GOAWAY handling and connection-max-age settings matter. Mitigations: bound connection lifetime on the client, use a headless Service with client-side load balancing for gRPC, or put a proxy/mesh in the path that balances per request. ## Cause 4: infrastructure-level intermittency Worth keeping in the differential: conntrack table exhaustion on busy nodes (`nf_conntrack: table full, dropping packet` in kernel logs) causes seemingly random drops; source-NAT port exhaustion for high-fan-out egress does the same; and for `externalTrafficPolicy: Cluster` on NodePort/LoadBalancer Services, an extra hop and SNAT can hide the true client IP and add a failure surface. These are less common than the shutdown race but produce a similar statistical signature. ## How to investigate systematically 1. **Quantify.** What percentage, and is it a round fraction of the replica count? Does it track deployments, scaling events, or nothing in particular? 2. **Correlate with pod lifecycle.** Overlay error timestamps with pod deletion and creation times. Errors clustered in the seconds after a delete are the race; errors evenly spread are a bad backend or infrastructure. 3. **Bypass the Service.** Curl each pod IP directly, in a loop, and compare. This separates "one backend is bad" from "the routing is racy". 4. **Check the shutdown path.** Does the app handle SIGTERM? Is there a preStop hook? Is the grace period long enough? Read the timings in `kubectl describe pod` for a terminating replica. 5. **Look at the client.** Connection pooling and HTTP/2 change the failure distribution; a client that retries idempotent requests on reset masks the problem entirely — which is also a legitimate mitigation. ## The design lesson Graceful shutdown in Kubernetes is a *distributed* operation: no single component knows the moment a pod stops being routable everywhere. The only reliable strategy is overlap — keep serving after you are told to stop, long enough for everyone else to find out.