You are setting the bake time for each stage of a canary rollout — how long the new version soaks at 1% before it advances to 10%. How do you decide that number, and why is a 10-minute bake worthless for some changes?

## Two clocks, not one People pick bake times the way they pick timeouts: a round number that felt reasonable once. There are actually two independent questions, and the bake must satisfy both. ### Clock one: statistical power A canary stage is a measurement, and a measurement needs samples. The relevant rate is not the service's request rate, it is the rate at which the *changed behaviour* is exercised inside the canary: ```text events_per_second = total_rps x canary_share x changed_path_share ``` At 1,000 rps, a 1% canary and a path that is 0.5% of traffic, that is 0.05 events per second — three per minute. A ten-minute bake yields thirty events. If the baseline error rate on that path is 0.1% and the regression takes it to 1%, thirty samples will usually show zero failures in both groups. The gate passes because the experiment had no power, not because the code is good. The consequence is a lever people forget: **raising the traffic share buys the same evidence faster than extending the bake**. Ten minutes at 10% gathers the same samples as one hundred minutes at 1%. Which one you choose is a blast-radius decision — more users exposed for less time, or fewer users for longer — and it should be made explicitly rather than by defaulting to a long bake at a tiny share. ### Clock two: the incubation period of the failure you fear Some failures are not sample-limited at all; they are time-limited. No traffic share makes them appear sooner, or the relationship is so weak that it may as well not exist: - **Resource accumulation** — a leaked object, an unbounded cache, a file handle never closed, a disk filling with logs. These surface when a threshold is crossed, hours in. - **Scheduled work** — an hourly reconciliation, a nightly batch, a weekly report, a monthly billing run. If the change touches that path, a bake that does not span the schedule tests nothing about it. - **Warm-state effects that only degrade later** — a cache whose eviction policy changed behaves fine while it is small. - **Expiry** — a token, a session, a lease or a certificate whose refresh path is the code you changed. The first refresh is the test, and it happens on the credential's clock, not yours. - **Diurnal position.** A bake that runs at 03:00 never sees peak concurrency. It tells you the version starts and serves; it says nothing about behaviour at the top of the day. The rule that survives contact with production: **bake at least one full period of the failure mode you are actually worried about**, and if that period is longer than any acceptable rollout, stop relying on the bake and use a different control — a load test at full volume, a synthetic trigger of the scheduled job, or an explicit watch window after the rollout with the change still attributable. ## What long bakes cost This is where judgement lives, because the costs are real and not always visible to the person choosing the number. - **A long mixed-version window.** Two versions serve simultaneously for the whole rollout. Every compatibility constraint — wire formats, cache entry shapes, shared state — must hold for that entire duration, and the longer the window, the more likely a rare cross-version interaction is exercised. - **Emergency changes queue behind it.** If a rollout occupies the pipeline for eight hours, the fix for an unrelated production bug waits, unless you have an express path that skips stages deliberately. Design that path *before* you need it, and require someone to own the decision to use it. - **Rollouts overlap.** If a rollout takes longer than your deploy interval, several are in flight at once. Attribution degrades: when something breaks, more than one change is mid-rollout. - **Batching pressure.** Teams facing a day-long rollout stop shipping small changes, and larger changesets are exactly what progressive delivery is trying to avoid. ## A shape that usually works Short, automated bakes at the smallest stages, sized by how long the automated check needs to gather its minimum samples — often minutes. Then a meaningful stage, say 10–25%, held long enough to cross a diurnal cycle and any scheduled job the change could touch — in practice, overnight. Then the fleet. The tiny stages catch gross breakage cheaply; the long soak catches the slow things; and only one stage in the sequence is paying the cost of a long hold. And say the quiet part in the design review: the bake time you can afford is a statement about how much risk you are accepting, not a technical constant. If the number was chosen because the pipeline felt slow, that is a decision about risk made by nobody.