skip to content

Using the Kubernetes Gateway API, how would you send 10% of production traffic to a canary version of a service while also allowing testers to reach that canary deterministically by sending a specific request header?

level: middleimportance: should knowfreq 38%

answer

  1. weights are relative shares, not percentages
  2. omitted weight defaults to 1 → accidental 50/50
  3. weight: 0 = wired but no traffic
  4. precedence: exact > longest prefix > method > most headers
  5. YAML order is not evaluation order

basics

~20 s

One HTTPRoute with two rules. The first rule matches the tester header and sends 100% to the canary Service. The second, catch-all rule lists both Services in backendRefs with weights 90 and 10. Header matches take precedence over the plain path rule.

solid answer

~50 s

Both behaviours live in a single HTTPRoute. **Weighted split:** one rule whose `backendRefs` lists both Services with `weight: 90` and `weight: 10`. Weights are relative, not percentages — the share is `weight / sum(weights)` — and a backend with `weight: 0` receives nothing. Distribution is per-request and stateless unless the implementation adds session affinity, so a browser session can bounce between versions. **Deterministic override:** a second rule with `matches: [{headers: [{name: x-canary, value: "true"}]}]` and a single `backendRef` to the canary. Header matches are `Exact` by default; `RegularExpression` is available as an Extended feature. Rule ordering is not positional. The spec ranks matches: exact path beats prefix, longer prefix beats shorter, then method, then the number of header and query matches. Because the header rule has strictly more match conditions, it wins over the plain catch-all. Progressive delivery is then just editing the two weights.

code

yaml · 23 lines
yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout
  namespace: shop
spec:
  parentRefs:
    - name: edge
      namespace: infra
  hostnames: ["shop.example.com"]
  rules:
    - matches:
        - path: {type: PathPrefix, value: /checkout}
          headers:
            - name: x-canary
              value: "true"
      backendRefs:
        - {name: checkout-canary, port: 8080}
    - matches:
        - path: {type: PathPrefix, value: /checkout}
      backendRefs:
        - {name: checkout-stable, port: 8080, weight: 90}
        - {name: checkout-canary, port: 8080, weight: 10}

go deeper

for a junior

Show the two-rule shape: a header-matched rule to the canary and a weighted catch-all, and know that weights are relative.

for a middle

Explain default weight of 1, weight 0, and that match specificity — not YAML order — decides which rule wins.

for a senior

Discuss rollout mechanics: per-version metrics, no session affinity by default, backward-compatible contracts, and mirroring as the zero-risk earlier step.

for a principal

Frame progressive delivery policy: what signals gate promotion, automated rollback, request-share versus user-share statistics, and whether this belongs in a route object or a delivery controller.

## Why this is a typed feature Under Ingress, canaries were annotation territory — `canary: "true"`, `canary-weight: "10"`, `canary-by-header` — with a second shadow Ingress object and semantics that differed per controller. Gateway API makes both mechanisms first-class fields on `HTTPRoute`, so the API server validates them and behaviour is comparable across conformant implementations. ## Weighted backendRefs A rule's `backendRefs` is a list, and each entry may carry `weight`. Key semantics: - Weights are **relative shares**, not percentages. `90/10` and `9/1` are identical; `3/1` is 75/25. The share is `weight / sum(weights)`. - The default when `weight` is omitted is `1`, so listing two backends without weights gives an even split — a common accident. - `weight: 0` means the backend receives no traffic but stays a valid, resolved reference. Useful for pre-warming or for keeping a rollback target wired up. - If every weight is `0`, the implementation should return a 500-class response rather than silently dropping. - If one backendRef fails to resolve (missing Service), that portion returns 500 while the others keep serving, and `ResolvedRefs` goes false with `BackendNotFound`. Selection is per-request. There is no sticky-session guarantee in the core spec, so a single user's requests can land on both versions. If a canary changes a client-visible contract mid-session, that matters; either keep changes backward-compatible or use an implementation offering session persistence. ## Header matching A `matches` entry may contain `headers`, each with `name`, `value`, and `type` defaulting to `Exact`. `RegularExpression` is an Extended-conformance feature, so check your implementation. Header names are case-insensitive per HTTP semantics; matching multiple headers in one entry is an AND. Multiple entries in the `matches` list are OR'd. You can also match `queryParams` and `method` the same way. This gives the deterministic path testers need: send `x-canary: true` and always reach the new version, independent of the random split. ## Precedence — why order in YAML does not matter A frequent misconception is that rules are evaluated top to bottom like an nginx config. They are not. The spec defines a total ordering so that different implementations agree: 1. Exact path match. 2. Prefix path match, longest prefix first. 3. Method match present. 4. Largest number of header matches. 5. Largest number of query-param matches. Ties across routes are broken by oldest `creationTimestamp`, then namespace/name, then rule order within a route. In this design, the header rule and the catch-all rule may share the same path prefix, but the header rule carries one extra header condition, so it ranks higher. That is what makes the override reliable rather than accidental. ## Operating a progressive rollout The workflow becomes a small sequence of edits: deploy the canary Deployment and its own Service; add it at `weight: 0` and verify with the header override; move to 1, 5, 25, 50; watch error rate and latency for the canary Service specifically; then either set the canary to 100 (or repoint the stable Service's selector) or drop the weight back to 0. Because it is declarative YAML, this is scriptable and GitOps-friendly, and progressive-delivery controllers drive exactly these fields. Two practical cautions. First, **observability must be able to distinguish the versions** — if both Deployments feed one dashboard, a 10% canary failing shows up as a small error-rate bump that noise can hide; label metrics by version. Second, **weights govern request share, not user share**; 10% of requests is not 10% of users, and heavy users skew the sample. ## Mirroring as a complement When you want production traffic against a new version without any user impact, use the `RequestMirror` filter instead. It copies matching requests to a second backend and discards the response, so the client is unaffected. Mirroring validates that the new version does not crash under real traffic shapes; weights validate that it serves users correctly. They answer different questions and are often used in sequence — mirror first, then weight.

  • A user reports being bounced between the old and new version within one session. Why, and what would you do?
    Core weighted routing selects a backend per request with no affinity, so consecutive requests from the same client can hit different versions. The durable fix is to keep the canary backward-compatible so bouncing is harmless. If it cannot be, use an implementation that supports session persistence, or gate the canary on a stable per-user attribute such as a cookie or user-id header rather than on weights.
  • How does a RequestMirror filter differ from routing 10% by weight?
    Mirroring copies matching requests to a second backend and discards its response, so users never see the mirrored version's output and errors there are invisible to them. Weighted routing actually serves a share of users from the canary, so its failures are user-visible. Mirror to check the new version survives real traffic; weight to check it serves correctly.

saying these in an interview costs you the question

  • Reading weights as percentages that must sum to 100
  • Omitting weight on one backend and expecting it to get zero traffic — the default is 1
  • Assuming rules are evaluated in YAML order like nginx location blocks
  • Expecting sticky sessions from weighted routing without any affinity mechanism
  • Treating a 10% request share as 10% of users when measuring canary health

context

resolving links…