A crop-yield model is scored weekly from imagery with a five-day revisit, so how often should its input shift test run?

## Cadence follows the data and the response, not the calendar Two questions set how often a shift test runs: 1. **How fast can the input actually change?** A national crop-yield forecast draws on overhead imagery with roughly a five-day revisit and on daily weather aggregates. Half the feature vector cannot move between Monday and Tuesday because no new observation exists. 2. **How fast could anyone act on the answer?** If the response to a confirmed input shift is to investigate an upstream feed and, at most, to schedule a retrain, a result produced six times between two scoring runs changes nothing that a single result would not. Running the test once per scoring cycle — weekly, here — matches both. Running it daily compares largely the same imagery against the same baseline six extra times, which does not detect anything earlier but does produce strings of correlated firings that read like six problems. ## Trigger on arrival, not on a clock A drift job scheduled at a fixed hour will eventually run while the feature pipeline is still writing. Half the regions are present, the national input distribution looks nothing like the baseline, and the test reports a large shift that is really a partially-written partition. The fix is operational, not statistical: - gate the run on a **completeness signal** from the feature pipeline (expected partitions present, expected row count within tolerance); - if completeness fails, **skip and report skipped** rather than testing the rows that happen to be there; - record which data version the run consumed, so a later reader can tell a real shift from an incomplete one. ## Window size follows from the smallest slice you intend to judge Cadence sets the detection window's span; the span has to leave enough rows in every cut you plan to test. With roughly 1.2 million fields scored nationally each week, a national test has more rows than any statistic needs. But once the test is cut by agro-climatic region, a small region may contribute only a few thousand fields, and a statistic computed on a couple of hundred rows swings wildly from run to run and manufactures firings. - Set a **minimum-row guard** and report "insufficient rows" rather than a number. - Where a slice is genuinely small but genuinely matters, give **that slice** a longer detection window (four weeks instead of one) and accept slower detection there. - Do not let the detection window overlap the reference window; if it does, the two sides share rows and the statistic is biased toward calm. | cadence | detection speed | statistic stability | firing volume | |---|---|---|---| | daily, on weekly-refreshed inputs | no faster in practice | worse, fewer new rows per run | much higher, and correlated | | once per scoring cycle | matched to when the input can change | good on national and mid-size slices | proportionate | | monthly | too slow to act within a season | very stable | low, but a break can run for weeks | ## Cadence multiplies with the slice and feature count The number of comparisons a run produces is features times slices. With 40 model inputs cut by 8 regions, one run is **320 tests**. At a 5% per-test false-positive rate that is about **16 red cells every run** before anything has actually shifted. Move that from weekly to daily and the same 16 arrive every day, which is how a drift dashboard becomes permanently red and stops being read. Three levers, used together: 1. run at the cadence the data justifies; 2. control the false-discovery rate across the run rather than applying a per-test cutoff independently (a Benjamini-Hochberg procedure at a stated rate, or a Bonferroni cut of 0.05/320 = 0.00016 if you would rather be conservative); 3. threshold on **effect size** — the magnitude of PSI or a distance — rather than on a p-value, since with hundreds of thousands of rows per slice almost any difference is significant. ## What this sounds like in a design round Good answers name the constraint before the number: the revisit interval, the scoring cycle, the response time, the smallest slice. Weak answers pick a cadence because it is a round number, or reach for the fastest possible schedule on the theory that more monitoring is more safety. On a weekly scoring pipeline it is not: it is the same evidence, resampled, with a higher chance that whoever is reading has stopped looking.