You own cluster DNS for a large, busy multi-tenant Kubernetes cluster. How do you make name resolution scale and stay reliable, and how do you decide how much capacity it needs?

## Framing: DNS is on every request path In a Kubernetes cluster, service-to-service calls, database connections, object-store access and outbound API calls all start with a name lookup. That makes cluster DNS a dependency of essentially every request, with no natural fallback. Two consequences follow: its availability objective must be at least as strong as anything that depends on it, and its failure modes must be *partial-tolerant*, because a partial DNS failure presents to teams as random unexplained errors everywhere at once — the most expensive kind of incident to triage. ## Build a capacity model, do not guess A usable model: ``` QPS ≈ active_pods × lookups_per_request × requests_per_second_per_pod × amplification_factor ÷ cache_absorption ``` - **lookups per request** is 1 only for applications that reuse connections; naive clients resolve per request. - **amplification_factor** is the killer. Under the cluster default of `ndots:5`, an external hostname walks the search list before the correct query, and the A and AAAA pair roughly doubles the packet count. Six-to-eight-fold amplification for egress-heavy workloads is ordinary. - **cache_absorption** is everything you deploy in front: node-local caches and in-process resolver caches. Measure the real numbers from the DNS server's own request metrics and the share of responses that are NXDOMAIN. A large NXDOMAIN share is direct evidence that the amplification term dominates, and it says the cheapest capacity win is reducing queries rather than adding replicas. ## Lever 1 — a node-local cache The highest-leverage change is a per-node caching resolver that pods query at a link-local address. It: - serves repeats from the node, removing the cross-node network hop and the address translation to a DNS pod; - absorbs the repeated negative answers from search-list expansion; - forwards upstream over TCP, which removes the UDP query-collision race that produces multi-second resolution stalls; - keeps working through brief central-DNS disruptions, converting a hard failure into a cache-warm degradation. Its cost is one more per-node component in the critical path, with its own upgrade and failure story — a node-local cache that dies takes DNS on that node with it, so it needs the same operational rigour as the kubelet. ## Lever 2 — caching policy Inside the DNS server, decide TTLs deliberately. Longer caching of external names cuts upstream traffic but slows propagation of real changes. Serving **stale entries** when the upstream is unreachable is usually the right trade for a platform: a slightly outdated address beats a resolution failure. Negative-answer caching bounds the damage from misconfigured clients hammering names that do not exist. ## Lever 3 — scale and protect the central deployment Only after caching should you buy capacity with replicas: - **Scale proportionally** to nodes or cores rather than pinning a fixed count, since clusters grow silently past their original sizing. - **Requests, not just limits.** A DNS server throttled at its CPU limit drops queries in bursts, which is the hardest symptom to attribute. - **System-critical priority class**, so it is never the thing evicted under node pressure. - **Anti-affinity across nodes and zones**, so no single node or zone failure takes a large share of capacity. - **A disruption budget**, so drains and maintenance cannot remove replicas below a safe level. - **A rollout that preserves capacity**: with a small replica count, a rolling update removes a large fraction at once. Surge and readiness gating must keep enough serving capacity throughout. ## Lever 4 — separate blast radii At scale, consider splitting responsibilities: one path resolves in-cluster names from the API-server-derived view, another handles forwarding to external resolvers. External resolvers fail, rate-limit and slow down for reasons entirely outside the cluster; when they do, you want in-cluster resolution to be unaffected. Keeping the two on separate instances or separate forwarding policies stops an outside dependency from taking down internal service discovery. ## Lever 5 — governance in a multi-tenant cluster Tenants can hurt shared DNS in three ways: applications that resolve on every request; workloads left at a default `ndots` while making heavy external calls; and workloads that bypass cluster DNS entirely by pinning their own resolver, which escapes node-local caching and any forwarding policy or visibility the platform relies on. The platform answers are admission-time: default resolver options per workload class, constrain or require review for pods that supply their own resolver configuration, and publish the expectation that clients reuse connections. Attribute query volume per namespace so the conversation with a heavy tenant is evidence-based. ## Verification and objectives Define what "working" means: a latency objective for resolution and a ceiling on server-failure responses, measured continuously. Then test the failure modes rather than assuming them — a game day that deletes replicas, blocks the upstream resolver, and drains a node running a cache, while watching what applications actually do. The findings are usually about clients, not servers: services that cache resolutions for their entire lifetime and keep dialling dead addresses, or that treat a resolution failure as fatal instead of retrying. ## The judgement to communicate Order matters. Reduce queries, then cache close to the client, then add capacity, then protect what you have from your own operations. Adding replicas first is the common and least effective response, because it scales the symptom rather than the cause — and a cluster whose DNS load is mostly failed search-suffix attempts is paying for capacity it did not need.