How would you design a drift alert on hourly sampled quality scores for a live assistant?

## What sampled online judging is for Offline evaluation tells you how a change performs on cases you already know about. Sampled online judging tells you how the system is doing right now, on real traffic, including the cases nobody anticipated. The mechanism is simple: take a small random slice of live sessions, score each one automatically, and treat the resulting time series as a continuous quality read. It fills the gap between explicit ratings (too sparse and too biased to trend) and human review (too slow and too expensive to run continuously). How the scorer itself is designed and validated — rubric shape, bias controls, agreement with human labels — is its own discipline. What matters here is treating its output as a monitored signal like any other. ## Choosing the sample Three parameters: rate, unit, and stratification. - **Rate.** Around 2% of conversations is a workable default: enough volume for hourly buckets on a high-traffic assistant, cheap enough to run indefinitely. Derive it rather than guess it — decide the smallest regression you want to catch and how fast, and check that the resulting bucket size gives that detection power. Low-traffic products often cannot support hourly buckets at all and should judge daily. - **Unit.** Sample whole sessions, not individual turns, if the failure you fear is conversational (looping, forgetting, failing to resolve). Sample turns if you care about per-response correctness. Mixing them makes the series uninterpretable. - **Stratification.** Pure random sampling under-covers small but important segments. Stratifying by intent or tier, then weighting back, keeps the headline number honest while giving each segment enough volume to alert on independently. Always sample *before* you know the outcome. Judging only sessions that already look suspicious produces a series that measures your filter, not your system. ## Building a baseline that survives seasonality A flat threshold ("alert below 0.80") fails immediately, because quality has structure. Overnight traffic differs from business hours. Monday differs from Saturday. A marketing push changes the intent mix for a day. A flat line either fires every night or is set so low it never fires. The usual fix is a **same-hour-of-week baseline**: compare this Tuesday 09:00 bucket against the distribution of previous Tuesday 09:00 buckets, and express the deviation in standard deviations of that history. That absorbs daily and weekly seasonality automatically. Keep several weeks of history, and exclude known-bad periods from the baseline or the incident you just had will normalise the regression it caused. ## Alert rules that do not cry wolf Hourly samples are small, so single-bucket alerts are dominated by noise. Practical shape: - Compute a **rolling mean over several hours** (six is a reasonable default) and compare that, not the raw bucket. - Fire when the rolling deviation crosses a threshold such as **1.5 sigma below baseline** and stays there. Requiring persistence across consecutive windows trades a little detection latency for a large reduction in false pages. - Add a **minimum-sample guard**: suppress the alert when the window's judged count falls below what the threshold assumes, or a quiet night will page you. - Alert **per segment as well as in aggregate**. A 20% regression confined to one intent that is 5% of traffic barely moves the headline number, and that is exactly the regression you want to catch early. ## The confounds that will bite you Three things move this series without quality changing: 1. **Scorer drift.** If the scoring model, its prompt, or its decoding settings change, the scale moves. Pin the model version and prompt hash, record both with every score, and re-baseline deliberately whenever you change them — never silently. Run the old and new scorer in parallel over the same window to translate between scales. 2. **Traffic-mix shift.** A campaign, an outage elsewhere, or a new channel changes which questions arrive. Aggregate quality drops because hard questions grew as a share, not because answers got worse. Record composition alongside the score and check it first when an alert fires; segment-level series usually settle the question immediately. 3. **Sampling changes.** Someone edits the sample rate or the stratification and the series steps. Version the sampling config the same way you version the scorer. ## What happens after the page An alert is only useful if it routes to a diagnosis. The runbook that pays for itself: check sample size and composition first; then compare segment series to localise; then pull the lowest-scoring judged traces from the window and read them. If they show a common failure mode, that is both your root cause and your regression-case candidate, which is how a drift alert feeds the eval loop rather than just interrupting someone's evening. ## Cost Judging is inference, so it has a bill. At a couple of percent of traffic it is typically a small fraction of serving cost, but the number should be explicit and monitored, and the sample rate should be a tunable knob — raised temporarily during a risky rollout, lowered when the series is quiet.