A saved query that selects cases by label in a TestRail-class case repository quietly returns fewer cases each release, though nothing was deleted. What causes that drift and how do you catch it?
answer
- the query is fine; the population moved
- new authors never learned the convention
- hyphen, plural, casing fork the value
- a shorter valid result is not an error
- count it every run and watch the trend
basics
~20 sNew cases are authored without the label, spellings fork into near-duplicates, bulk edits drop values, and the query keeps returning a valid, shorter set. Catch it by recording the result count every run and reconciling the selection against the same area counted another way.
solid answer
~50 sThe query has not changed; the population it runs over has. The usual causes compound: cases authored after the convention was set never get the label, an author types a near-duplicate spelling that no longer matches, a bulk edit overwrites labels rather than adding to them, and a permission boundary hides part of the result from the account running it. None of these produce an error — a query that matches fewer rows is a perfectly valid query. The fix is in two parts. **Detection**: store the selected count with every run and alert on a step change, and periodically reconcile the label-driven set against the same area selected on a mandatory attribute, watching the gap. **Prevention**: if a selection must be complete, it should not depend on a free label at all — promote the dimension to a controlled mandatory field and backfill.
go deeper
Understand that a saved query stores the question, not the result, so it returns whatever currently matches — and that matching fewer cases is not treated as an error by anything.
Explain the concrete causes: unlabelled new cases, forked spellings, a replacing bulk edit, and permission filtering, and say which of them produce a step change versus a slow slide.
Demonstrate the investigation: record the count per run, reconcile against a mandatory attribute for the same area, audit distinct label values, and re-run as a fully-visible account to separate absence from invisibility.
Own the standing rule about which selections may depend on a free label at all, who owns the label vocabulary, and what promoting a label to a mandatory field obliges you to backfill.
## The query did not change; the repository did A saved query is a stored question, re-evaluated every time it is used. That is its great strength — and the reason it can decay without anybody touching it. The definition on screen is exactly what somebody wrote and approved. What changed is the set of cases it runs over, and a query returning fewer rows than last month is not an error condition. It is a correct answer to the same question about a different population. ## Where the cases go **Authoring gap — by far the biggest cause.** The label was a convention agreed in a meeting some releases ago. Everyone who was there applies it. Everyone who joined since does not, because nothing in the authoring form asks. The repository keeps growing, the labelled subset stops growing, and the selection covers a steadily smaller fraction of the area it claims. This decays continuously rather than in a step, which is exactly why nobody notices. **Spelling fork.** A free label is typed. `smoke-critical` and `smoke_critical` are one idea to a human and two values to a query. Type-ahead helps only the author who begins typing the same characters, so plurals, hyphens and casing quietly split the vocabulary. The cases still carry a label — just not the one the query asks for. **Bulk-edit overwrite.** Somebody selects a subtree and sets labels to standardise them. If the operation replaces the label set rather than adding to it, every unrelated label on those cases is gone in one action, including the one your suite depends on. This produces the step change, and it is the version that is at least detectable if you are watching the count. **Visibility, not absence.** Permissions typically attach to the folder tree, so an account without read access to a subtree receives a shorter result. The cases match; the account cannot see them. Two people run the same saved query and get different counts, which is a confusing signal until you know to look for it. **Lifecycle removal.** A case retired from the active set stops appearing in queries scoped to active cases. That is usually intended, but it moves the count, so it belongs in the list of things to rule out before assuming something is wrong. ## Detection: make the silent number loud The common thread is that every cause produces a *valid, smaller* result. So the countermeasure is not error handling — it is trend. 1. **Record the selected count with every run.** One number, stored beside the run. A step change means a bulk edit or a permission change; a slow slide means the authoring gap. 2. **Reconcile against a second axis.** Count the cases in an area by a mandatory controlled attribute, and compare with the label-driven count for the same area. The ratio is a coverage figure. A widening gap is the authoring gap made visible, and it appears months before a defect escapes. 3. **Audit the label vocabulary itself.** List the distinct label values and eyeball them for near-duplicates. Values differing by a hyphen, a plural or casing are almost always one intended value that forked. 4. **Run the query as a maximally-visible account** at least once when investigating a discrepancy, so you can separate *not matched* from *not visible*. ## Prevention: stop depending on a label for completeness Detection tells you when the set is wrong. Prevention stops the dependency: - **Promote the dimension.** If a selection must be complete, the thing it selects on should be a **mandatory controlled field**, not a typed label. A mandatory field cannot be skipped by a new author, cannot be misspelled, and can be audited for a default-value pattern. - **Backfill when you promote.** A new mandatory field only constrains new and edited cases. Until the existing thousands are populated, the field is a label with more ceremony. - **Prefer derivation over opt-in.** A selection derived from attributes a case must answer anyway (`component`, `type`, `priority`) has no separate marking step to forget. Reserve an explicit opt-in label for tiers where membership is a genuine judgment call rather than a consequence of what the case is. - **Give the vocabulary an owner.** Somebody should be able to retire a dead label and merge a forked one. Without that, the tail only ever grows. ## The judgment to carry away A shrinking result set is the quietest failure a case repository produces, because every component of it is behaving correctly. The repository stored what it was told, the query matched what it was asked, and the run passed over what it received. Only the trend across releases carries the signal — which is why the count belongs in the run record from the first day, not from the day after it cost you something.
- Two people run the same saved query and get different counts. Where do you look first?Read permissions, which usually follow the folder tree rather than the attributes. The query matches the same cases for both accounts and the result is then filtered to what each may see, so a user without access to a subtree gets a silently shorter list. Re-run it as an account that can see the whole repository to establish the true count.
- How do you merge a label that has forked into several spellings without breaking anything?List the distinct values and agree which one survives, then bulk-apply the survivor before removing the variants, so no case is ever unlabelled in between. Use an additive edit rather than a replacing one, or you will drop unrelated labels on the same cases. Afterwards, if the dimension mattered enough to fork, it probably deserves a controlled field.
saying these in an interview costs you the question
- Assumes a shrinking result set would raise an error
- Blames the query rather than the population it runs over
- Ignores that permissions can shorten a result silently
- Uses a replacing bulk edit to standardise labels
- Relies on a free label for a selection that must be complete