Which evidence should a team carry on every job run, and which should it capture only on a deliberate re-run?

## Two kinds of evidence, two kinds of cost Every pipeline pays for its own diagnosability twice, and the two payments look nothing alike. - **Always-on evidence** is paid on every run, forever, whether or not anything goes wrong. A few integers per step are free at any scale. A trace per record is a second pipeline. - **Capture-on-demand** is paid only when invoked, but it buys a *future* observation, not the one you missed. It is worth exactly as much as your ability to make the failure happen again. The decision is therefore not "how much telemetry do we want" but "which observations, if we do not make them now, can never be made". ## What makes a later capture impossible Before relying on re-running with more capture, check whether the re-run can exist: 1. **The input has moved on.** A source that retains a window will have discarded the records by the time anyone looks. Whether the input can be re-read at all is its own subject, but its answer decides whether this strategy exists. 2. **The conditions are gone.** The cluster width, the machines you were given, the other work sharing the pool and the reclaimed host that triggered the recomputation are not reproducible by asking nicely. 3. **The failure is not deterministic.** Anything caused by interleaving, by placement or by a machine's own trouble may simply not happen twice. 4. **The code has moved.** By the time a wrong number is noticed downstream, the logic that produced it may be two deployments old. Where any of these holds, the evidence must have been carried, or it does not exist. ## A defensible default split | Evidence | Cost per run | Reconstructible later? | Verdict | |---|---|---|---| | A few totals per step, named by condition | Negligible | No, not for the run that failed | Always | | Run identity, width, start and end, per-unit status | Negligible | No | Always | | A bounded sample of rejected records | Small and fixed | Sometimes | Always, capped | | Per-record traces through every step | Large, on every run | Usually yes | On demand | | Full payloads of every rejected record | Large and sensitive | Sometimes | On demand | | A counter per grouping key | Unbounded in the summing process | Yes | Never | The first three rows share a property: they are the same size whether a run handles a thousand records or a trillion, and they describe the run that actually happened. The middle rows scale with the data and can usually be recovered by asking again. The last row is the standing temptation that quietly makes the coordinating process the fragile part of the system. ## What a mature team deliberately stops measuring There is a real cost to keeping everything, and it is not only money: - **Attention.** Evidence nobody has read in a year still has to be understood by whoever finds it during the next incident. - **Sensitivity.** Captured records accumulate whatever personal data the input carried, in a place with weaker handling than the pipeline itself. - **Change friction.** Instrumentation nobody uses is instrumentation nobody maintains, and it breaks silently at the worst moment. So pruning is part of the discipline: retire counters nobody has queried, keep the vocabulary small enough that an unfamiliar pipeline is legible, and prefer four totals everyone understands to forty that only their author does. ## The switch has to be exercised A capture-on-demand strategy rests on a path that, by construction, is almost never used. The failure is predictable: - it needs a code change and a deployment, so it takes a day at the moment it is needed in an hour; - nobody remembers where the raised output lands, or it lands somewhere with no capacity; - it was written against an older shape of the job and no longer compiles; - it raises capture on a run that can no longer read the input that failed. Run it periodically on an ordinary day, on a real pipeline, and confirm the evidence arrives where the runbook claims. ## The argument to make at the top The honest framing is not a dashboard question, it is a promise. Carrying no evidence is cheapest until the first unexplained number, and the cost of that number is absorbed by the people who trusted it, not by the team that owns the job. So ask how long an unexplained wrong figure may stand before someone must be able to say why — a day, a week, never — and buy exactly the always-on evidence that answers within that window. That number is a service promise, it belongs in the same conversation as freshness and availability, and it is the only sound basis for deciding what the pipeline pays for on every single run.