How do canary rollouts and A/B tests differ in what they control for?

## Two mechanisms, two questions Both a canary and an A/B test route some live traffic to a new variant, which is why teams blur them. The questions they answer are different in kind, and a change usually needs both. **Canary rollout asks: will this hurt anyone?** It is a *risk-control* mechanism borrowed from operational deployment practice. Expose a small slice, watch, ramp if clean, roll back instantly if not. Its virtue is that the cost of being wrong is bounded by the exposed fraction and the time you sat there. **A/B testing asks: does this help, and by how much?** It is a *causal-inference* mechanism. Randomization makes the arms comparable so that the difference between them can be attributed to the change; sample size determines the smallest effect you can resolve; a pre-declared primary metric prevents you from finding a winner after the fact. One bounds damage. The other produces a number you can defend. ## Why 1% cannot answer an A/B question The smallest effect an experiment can detect scales with the noise in the metric and inversely with the square root of the sample size in the *smaller* arm. A grocery site's conversion rate is noisy at the daily level; resolving a half-percent relative change takes a lot of sessions. A 1% arm accumulates those sessions roughly fifty times slower than a 50/50 split does, which turns a one-week read into an impractical one. So the metrics a canary can genuinely act on are the ones with large, fast, obvious signals: error rate going from near-zero to non-zero, p95 latency doubling, cost per request tripling, a safety classifier firing at all, output-format validity collapsing. Those show up in hundreds of requests. Conversion does not. The corresponding mistake is the mirror image: running a 50/50 split on a change that has never been exposed at all. If it is broken, you broke it for half your users, and you find out only when the metrics come in. ## What a canary actually controls Beyond sample-size logic, a canary controls things a randomized split does not. **Blast radius over time.** The exposed population is small *and* the exposure is reversible in minutes. That combination is what makes it safe to try a structural change on production at all. **Coverage of load patterns.** A canary is held for a duration, not just a fraction, precisely because failures are often time-dependent — a weekend peak, a nightly batch, a Monday-morning burst, a cache that only fills after hours. Holding 1% for 48 hours across a weekend peak before ramping is a deliberate choice to see the system under a load shape a Tuesday-afternoon test never produces. **Staged confidence.** Each ramp step is a new observation at higher volume. Failure modes with low per-request probability become visible at 25% that were invisible at 1%, so the ramp is itself a sequence of tests rather than a formality. ## What an A/B test controls **Confounding.** Seasonality, marketing pushes, an outage, another team's launch — all hit both arms simultaneously, so a difference between arms is attributable to the change. A before/after comparison, which a canary superficially resembles, has no such protection: the world changed between the two periods too. **Selection.** Random assignment ensures the arms have comparable users. Canary slices are often *not* random — routed by region, by pod, by client version, by internal-user flag — which makes them poor comparison groups even when volume would suffice. **Multiple comparisons and peeking.** A disciplined split pre-registers the metric and the stopping rule. Repeatedly checking an experiment and stopping when it looks good inflates false positives, which is why sequential testing methods exist for teams that need to watch continuously. ## How they compose The usual release path for a meaningful LLM change: offline gate, then shadow on mirrored traffic where side effects allow, then a small canary held long enough to cover a real load cycle, then a powered randomized split on the primary metric with guardrails, then a staged ramp to full exposure. Note that canary and A/B can overlap physically — a 1% canary arm *is* randomized in many implementations, and its data can contribute to the eventual analysis. The distinction is not the plumbing, it is what you are entitled to conclude. Nothing breaking at 1% licenses more exposure; it does not license the claim that the change is better. ## Judgment calls a lead owns **When to skip the split entirely.** Some changes are not worth a powered experiment: a latency optimization with no expected quality change, a security fix, a cost reduction at measured parity. Canary plus guardrails is sufficient, and demanding a full A/B on everything is its own organizational cost. **When a split is impossible.** Small B2B populations, strong network effects between users, and changes that alter shared state often cannot be cleanly randomized at the user level. Then the toolkit shifts — switchback designs, cluster randomization by account or region, interrupted time series — each with weaker inference that you should name honestly rather than pretend a user-level split existed. **How long to hold.** Long enough to cover the load and behaviour cycles that matter, and long enough for novelty and disruption effects to decay. Both arguments push toward days, not hours, for user-facing changes. **How much of the release budget experimentation deserves.** Every powered experiment costs calendar time and holds a fraction of users on the worse variant. A team that A/B-tests every prompt tweak ships slowly; a team that canaries everything and measures nothing accumulates unverified changes. Setting that policy — which classes of change require inference and which only require safety — is the principal-level call. ## What interviewers listen for The expected answer separates risk control from inference and explains why sample size makes a canary structurally unable to settle a fractional business-metric question. Strong candidates add the non-random-slice problem, the role of hold duration, and the cases where randomization is not available at all.