Describe how a canary deployment progressively shifts traffic to a new service version, and what signals should automatically trigger a rollback.

## How the traffic split works A canary deployment is a progressive-delivery technique where a new version of a service is exposed to a small, controlled slice of real production traffic while the majority continues to be served by the known-good ('stable' or 'baseline') version. Mechanically this requires a **traffic-splitting layer** that can route by percentage: - a service mesh sidecar (Istio, Linkerd) - an API gateway - a load balancer with weighted target groups - Kubernetes-native tooling (Argo Rollouts, Flagger) that manipulates the ratio of pods behind a Service A typical rollout starts at a low weight, say 5%, holds there for an observation window, then steps up through stages — 5% -> 25% -> 50% -> 100% — each gated by whether the canary is healthy. ## Why the technique exists Canary exists because no staging environment fully reproduces production traffic shape, concurrency, and data skew; the only way to truly validate a change under real conditions is to expose it to some real traffic, while capping the blast radius so a bad release only affects a bounded fraction of users rather than everyone at once, as a blue-green cutover would. It sits between blue-green (instant, all traffic) and rolling deployment (no explicit control group) by keeping an explicit baseline running the whole time for direct comparison. ## The trade-off The core trade-off is speed versus safety. - **Slower than blue-green.** Canary rollouts are slower than blue-green — a full promotion can take tens of minutes to hours because each stage needs a genuine observation window, and rushing it defeats the purpose (some bugs, like a slow leak, only show up under sustained load). - **Operational sophistication.** Canary also demands operational sophistication: per-version metrics tagging, a traffic-splitting mechanism, and ideally automated analysis (Flagger, Argo Rollouts with a metrics provider), because manual dashboard-watching doesn't scale and misses slow-burning regressions. - **Another cost:** for low-traffic services, a small canary percentage may not get enough requests to be statistically meaningful. ## Automated rollback triggers Automated rollback triggers should compare canary metrics against baseline metrics over the same window rather than relying on absolute thresholds alone, since absolute thresholds don't account for background noise. Common signals: - **error rate** crossing a delta versus baseline - **latency percentiles** (p95/p99) regressing beyond a margin - **saturation** (CPU, memory, queue depth, pool exhaustion) - for higher-stakes services, **business-level signals** (checkout completion rate, payment failure rate) When a threshold trips, automation should immediately shift weight back to 0% rather than waiting for a human, because minutes of exposure at scale can mean real customer impact. ## Failure modes 1. A key failure mode is **'metric dilution'**, where the canary's bad signal gets averaged into an aggregate dashboard alongside the much larger baseline traffic and never crosses an alert threshold — which is why per-version tagging in metrics is mandatory, not optional. 2. Another is **sticky-session or cache-affinity bugs**, where a canary that looks fine in aggregate is actually failing consistently for a subset of pinned users. 3. A third is a **stateful or migration-carrying canary** — if the new version writes data in a new format or runs a destructive migration, even a 5% canary can corrupt data that all versions, including the eventual rollback target, now depend on; canary is far safer for stateless, backward-compatible changes than for schema-breaking ones. ## Where it shows up A well-known real-world pattern: **Netflix** pioneered automated canary analysis (Kayenta, later part of Spinnaker, statistically compares canary and baseline metrics and judges pass/fail automatically); many teams today use lighter-weight equivalents like **Flagger** on Kubernetes, automating the same stepped-traffic-plus-metrics-gate loop against Prometheus metrics.