Your handheld target list has not changed in two years. How do you decide what it should run on now?

A target list that has not moved in two years is not stable, it is unowned. The field underneath it has moved on all three axes — version lines, capability tiers, screen classes — and the question is what evidence should move the list, and how often you are willing to pay for moving it. ## Two decisions, not one - **What the product supports** is a commitment to users about which devices and versions will work. That is decided outside the suite, and it changes rarely and deliberately. - **What the suite runs on** is a subset of that commitment chosen for information value and cost. It can and should change more often. Confusing the two produces both failure modes at once: teams add every supported combination to the run list because "we support it", and teams refuse to touch the run list because they think changing it changes the promise. Keep them separate and each becomes tractable. ## The evidence that should move a row | Signal from the field | What it argues for | The caveat | | --- | --- | --- | | A version line's usage share has fallen across two windows | move it from every run into a periodic wide pass | share can be seasonal; look at a trend, not one week | | A new version line is appearing and climbing | add it before it is the majority, not after | early adopters are unrepresentative of who follows | | A capability tier is over-represented in real failure reports | strengthen that row, or split it | a spike can be one bad build rather than the tier | | A row has produced no unique failure in a year | it is a candidate for lower frequency | check whether it ran the cases that could have failed | | Displays of a new geometry are shipping in volume | add a screen row that crosses the layout switch | geometry only needs a row if an arrangement actually changes at it | Two things are worth naming explicitly. First, weight by **real usage, not device count** — sessions, or whatever unit reflects exposure — because a row that is 2% of devices and 15% of activity is not a 2% risk. Second, **popularity and information are different arguments.** A hugely popular row that behaves identically to another row in the list adds cost and no signal; you keep it for the reassurance, not for the coverage, and you should know which of those you are buying. ## Cadence or trigger There are two defensible policies and the conditions pick between them: 1. **Fixed cadence** — re-cut every two or three releases regardless. Cheap to run, predictable, and it forces someone to look. It suits a product whose field moves smoothly and whose team would otherwise never revisit the list. 2. **Stated threshold** — re-cut when a signal crosses a line you wrote down in advance (a row drops below some usage share for two consecutive windows; a new line passes some share). It suits a field that moves in jumps, such as a market where one device family dominates, but it needs the thresholds agreed before the data arrives, or the argument becomes about the threshold instead of the row. What is not defensible is re-cutting opportunistically after each embarrassing escape, because that ratchets the list upward forever: rows are added by incident and never removed by evidence. ## What a re-cut costs Changing the list breaks comparability. Pass rate, duration and flakiness trends are only meaningful across the same targets, so a revision resets the baseline you were reading. The mitigation is structural rather than heroic: - Keep a **small fixed core** that changes rarely, and let the remaining rows rotate through a periodic wide pass. Trends live on the core. - **Version the list** with a revision number and the date and window of the evidence behind it, and record which revision produced each set of results, so a change in pass rate can be attributed to the list rather than blamed on the product. - Record **why each row exists** — the axis it covers and the failure class it is there to catch. A row whose reason nobody can state is the cheapest thing in the list to remove. ## The axis that quietly stops being true Version lines announce themselves and displays are visible, but capability drifts silently. A row labelled "low tier" that was chosen three years ago may now be an ordinary mid-range device: it still runs, it still passes, and it no longer produces the memory-pressure and thermal failures it was added for. Re-cutting means re-checking what each label denotes, not just re-checking the shares next to it.