skip to content

An authoritative DNS server answers 5% of queries for api.example.com with a canary address; why does that not mean 5% of users reach the canary?

level: middleimportance: should knowfreq 34%

answer

  1. who sends the queries
  2. a draw per cache miss
  3. one draw, whole resolver population
  4. no weight field in A records

basics

~20 s

The 5% applies to queries reaching the authoritative server, which are recursive resolvers' cache misses. Each drawn answer is cached for its TTL and served to every client of that resolver, so exposure follows resolver populations, not users.

solid answer

~40 s

Weighted answers are steering logic in the authoritative server: for each query it draws stable or canary in proportion to the configured weights; `A` and `AAAA` records carry no weight field. But the server sees queries from recursive resolvers, not users, and a resolver queries only when its cache is empty. So each draw is cached for the TTL and handed to every client of that resolver. If one resolver serves 40% of your users and draws the canary, about 40% of users hit it until the TTL runs out. Averaged over many resolvers and TTL periods the share approaches 5%, but minute by minute it swings widely, a user can flip between versions, and removing the canary only affects new cache fills.

go deeper

for a junior

Remember that a weighted answer is picked by the authoritative server, and that caches share whatever it picked.

for a middle

Walk through a cache miss, the draw and the TTL, and show with numbers why one large resolver can dominate the canary's exposure.

for a senior

Explain what you would measure at the canary itself, how you would size the weight and TTL, and why rollback is not instant.

for a principal

Argue when DNS-level weighting is good enough for a rollout and when the split must move to a layer that sees individual users.

## How a weighted DNS answer is produced A **weighted answer** is a decision the authoritative server makes each time a query arrives. The operator configures candidates with weights, for `api.example.com` a stable address `192.0.2.20` with weight 95 and a canary address `198.51.100.20` with weight 5, and the server's steering logic picks one candidate with probability weight divided by total weight. It then returns the chosen address as an ordinary answer. Two points are easy to miss: - **Plain `A` and `AAAA` records have no weight field.** Weighted answers are behaviour of the authoritative server implementation. The recursive resolver receives an ordinary answer and cannot tell it was chosen by lot. - **Where DNS does standardize a weight, as in `SRV` records** (RFC 2782), the *client* chooses among targets of equal priority, in proportion to their weights. Weighted `A`/`AAAA` answers move that choice to the server. ## What the weight actually counts The authoritative server almost never sees users directly. It sees queries from recursive resolvers, and a resolver queries only when its cache has no fresh copy. So the 5% applies to **cache fills**, not to users: 1. A resolver misses its cache for `api.example.com` and queries the authoritative server. 2. The server draws: 95% stable, 5% canary. 3. The resolver caches that one answer for the record's TTL. 4. Every client of that resolver during the TTL gets the same address. 5. When the TTL expires, the next client query triggers a fresh draw. ## A worked example Suppose one large resolver serves 40% of your users, 200 small resolvers serve the other 60%, and the answer's TTL is 60 seconds. - Roughly once a minute the large resolver refills its cache. Each time, with probability 0.05, it draws the canary, and then **about 40% of all users** reach the canary for up to a minute. - The small resolvers draw independently, so across them the canary share hovers near 5% of their 60%, about 3% of all users. - Averaged over hours, canary traffic approaches the configured 5%, but minute by minute it swings between about 3% and more than 40%. A canary meant to cap the blast radius at 5% can therefore expose nearly half the users at once. The more concentrated your users are behind a few resolvers, the larger the swings. ## Other things that move the share - **EDNS Client Subnet.** A resolver that sends ECS (RFC 7871) may cache separate answers per client prefix. That means more draws and a smoother split for its users. - **Hosts and applications** can hold an address longer than the DNS answer lives, and a connection opened to the canary stays there until it closes. - **Client fallback.** A client can retry another address when the canary fails only if the answer contained one; a single-record weighted answer leaves nothing to fall back to. - **Traffic per user is uneven.** A few heavy clients behind one resolver outweigh many light ones. ## What weighting can and cannot give you | Goal | Weighted DNS answers? | Why | |---|---|---| | Rough share over a long window | Yes | Draws average out across resolvers and TTL periods | | Hard cap on exposed users | No | One draw at a large resolver exposes its whole population | | Keep one user on one version | No | Each cache refill is an independent draw | | Instant rollback | No | Withdrawing the canary affects only new cache fills | How long cached answers survive, and the trade-offs of short versus long TTLs, is the caching topic's subject. For this question the TTL matters only as the length of each draw. ## Running a DNS canary honestly - Measure exposure at the canary endpoint, in requests and distinct clients, rather than trusting the configured weight. - Keep the TTL short enough that one unlucky draw does not last long, accepting more queries at your authoritative servers. - Pick a weight whose worst case, the largest resolver population drawing the canary, is still acceptable. - If you need per-user stickiness or a precise percentage, split traffic at a layer that sees individual users, which DNS does not.

  • How would EDNS Client Subnet change the canary exposure of a large recursive resolver's users?
    With ECS the resolver sends a truncated client prefix and may cache a separate answer per prefix the authoritative server scopes. The server then draws once per prefix instead of once for the whole resolver, so a single canary draw exposes only that prefix's clients and the overall split gets smoother. Resolvers that do not send ECS still get one draw for everyone.
  • How do SRV record weights differ from weighted A record answers?
    SRV weights are a field in the record (RFC 2782). The resolver returns every SRV target, and the client picks among targets of equal priority in proportion to their weights, so each client makes its own draw. Weighted A answers have no field at all: the authoritative server draws, and the cache shares the result.

saying these in an interview costs you the question

  • The weights guarantee that exactly 5% of users reach the canary
  • Recursive resolvers apply the weights separately for each client
  • A and AAAA records carry a weight field for this
  • Deleting the canary record stops canary traffic immediately
  • Weighted DNS answers keep each user on one version