Workloads in a cluster intermittently fail name resolution: most lookups succeed, but a small fraction time out after roughly five seconds. How would you investigate, and what causes are plausible?

## Read the number first A lookup that fails at about five seconds is not a slow server. The standard C library resolver waits a fixed timeout — five seconds by default — then retries or gives up. So the query, or its answer, never arrived. That single observation eliminates "the upstream resolver is slow" as the primary hypothesis and points at packet loss or a saturated server dropping requests. ## Step 1 — scope the failure Gather cheaply before theorising: - **Which names?** Only external names failing suggests the search-list walk and upstream forwarding are involved. Cluster names failing too suggests the DNS pods themselves. - **Which nodes?** Concentration on a few nodes points at node-local causes: a saturated connection-tracking table, an unhealthy DNS pod scheduled there, or an overloaded node. - **Which clients?** A single application failing while others are fine often means that application does the most lookups — no caching, new resolution per request. - **When?** Correlate with deploys, node scale-ups, or a rollout of the DNS Deployment. ## Step 2 — the server side CoreDNS is a Deployment with a small default replica count that many clusters never revisit as they grow. - **Capacity.** Are two replicas serving thousands of pods? Check whether CPU throttling is occurring against the container's limit — a throttled DNS server drops queries in bursts, which produces exactly this intermittent pattern. - **Health.** Restarts, OOM kills, or evictions. If the DNS pods have no resource requests and no priority class, they compete with application pods and can be evicted under pressure. - **Metrics.** Request duration, and responses broken down by response code. A high NXDOMAIN share is the fingerprint of search-list amplification: the server is spending its capacity answering the expansion attempts for external names. - **Upstream.** Cache misses for external names are forwarded; if the upstream resolver is slow or rate-limiting, forwarded queries queue and time out. Compare in-cluster name latency against external name latency to separate the two. - **Isolate a replica.** Query a specific DNS pod's address directly rather than the Service address. If one replica misbehaves and others are fine, you have found it; if all are fine when queried directly, suspect the path to them. ## Step 3 — the client side Read the effective `/etc/resolv.conf` in an affected pod. With `ndots:5`, every external hostname triggers several suffix attempts, each usually issued as a parallel A and AAAA pair. That multiplies both server load and the number of chances any single query is lost. An application that resolves per request rather than reusing connections multiplies it again. ## Step 4 — the datapath race The well-known cause of precisely-five-second stalls: a resolver sends the A and AAAA queries from the **same source socket** at nearly the same instant, to the DNS Service address. Both packets need a connection-tracking entry for the translation to a backing DNS pod, and concurrent creation of two entries for the same tuple can fail, so one packet is dropped. The application then waits the full resolver timeout. Corroborate with `conntrack -S` on affected nodes — insert-failed counters climbing in step with the incidents — and by checking whether the connection-tracking table is near capacity, which produces its own drops. Workarounds, weakest to strongest: resolver options that stop issuing the two queries simultaneously on one socket; disabling AAAA lookups where the workload is v4-only; and the structural fix, a node-local cache that answers from the node and forwards upstream over TCP, taking the racing UDP pattern out of the picture. ## Step 5 — the fixes, in order of leverage 1. **Node-local DNS cache.** A per-node cache pod that pods query at a fixed local address. It removes most cross-node DNS traffic, absorbs the repeated NXDOMAIN answers, and uses TCP upstream, which eliminates the UDP race. It is the standard remedy at scale precisely because it requires no application change. 2. **Right-size and protect the DNS Deployment.** Scale replicas with cluster size (an autoscaler proportional to nodes or cores is common), set requests so it is not throttled, give it a critical priority class, add anti-affinity so replicas do not share a node, and a disruption budget so maintenance cannot take them all. 3. **Reduce query volume.** Lower `ndots` for egress-heavy workloads, fully qualify hot external names, and fix applications that resolve on every request instead of reusing connections. Raising the cache TTL for external names helps too. 4. **Watch the rollout path.** Rolling the DNS Deployment briefly removes capacity; with only two replicas that is a large fraction. Ensure readiness gating and surge behaviour keep enough serving capacity during upgrades. ## How to present this in an interview Lead with the meaning of the five seconds, then show the layered narrowing — names, nodes, clients, time — then name both plausible causes (server saturation and the query race) and say how you would distinguish them with evidence. Finish with the node-local cache as the change with the best leverage, and the capacity and protection work that stops it recurring.