Two teams' stream pipelines have now died from silent buffering; what would you require of every pipeline before it ships?

## What the standard has to guarantee After two outages with the same root cause, the goal is not to ban a construct. It is to make one class of failure impossible to reach by accident: **a pipeline whose memory is bounded only by the machine, discovered at the moment the machine runs out**. Everything in the standard should serve that, and anything that does not is process for its own sake. The framing that makes it enforceable: retention is a **budget**, and a budget that nobody wrote down is a budget nobody can review. ## The four requirements 1. **No unbounded retention by construction.** Every stage that can hold items declares a capacity and what happens when it is full. The choice of what to sacrifice when full is deliberately left to the team, because it depends on whether a stale item is still worth anything — but making *some* choice is mandatory. The value of the rule is not the number chosen; it is that a human chose it at design time instead of the allocator choosing at 3 a.m. 2. **Capacity derived from the freshness budget, checked against memory.** Two bounds apply and the smaller wins: `capacity x bytes per item` must fit the memory the process can spare, and `capacity / drain rate` must be under the delivery age consumers need. In practice the age bound is usually far tighter, which is why sizing from spare memory produces pipelines that survive and deliver hours-old data. 3. **Depth and delivery age exported per retaining stage.** Both, not one. Depth without a drain rate cannot be interpreted; delivery age states the harm directly. The alert fires on **depth that fails to return to baseline across successive periods**, because a burst pushing depth up is normal and an absolute threshold either fires constantly or never. 4. **Evidence before the first deploy.** A soak run with a deliberately induced sustained deficit, showing flat memory, a bounded delivery age that matches the declared capacity divided by the drain rate, and the full-hold policy actually firing. A run that never reached the bound proves nothing about what happens at the bound. ## Deriving capacity, worked | Input | Value | Bound it gives | |---|---|---| | Retained bytes per item | 400 bytes | — | | Memory the process can spare | 3 GB | about 7,500,000 items | | Drain rate of the slowest stage | 11,500 items per second | — | | Delivery age the consumers need | 60 seconds | 690,000 items | | **Declared capacity** | | **690,000 — the tighter bound** | The table is the artefact the review asks for. It is five numbers, it fits in a pull request description, and producing it forces the team to know its item size, its drain rate and who consumes the output — three things that are usually unknown in exactly the pipelines that fail. ## The audit that goes with it Declared holds are the easy half. The standard also asks each team to list its **implicit** retention points and say what bounds each: a time-based grouping stage bounded by input rate times window length, a flattening stage bounded by its concurrency limit or by nothing at all, a de-duplicating stage bounded by the cardinality of its key space, a handoff between workers bounded by its queue capacity. Any line of that list whose bound is proportional to the input rate has to be converted to a constant before the pipeline ships. ## What you deliberately do not mandate - **One overflow policy for everyone.** Whether to keep the oldest, the newest, or to fail depends entirely on what the data is for, and a central mandate would be wrong in half the cases. - **One capacity number.** Rates and item sizes differ per pipeline; a shared constant is a number that is wrong everywhere. - **A specific implementation.** The standard is about declared bounds and exported signals, which every reasonable implementation can provide. ## The cost you are accepting Bounded pipelines fail **more often and earlier**. A condition that used to be invisible for four hours now produces an explicit event within minutes, sometimes including deliberate loss. That is the trade: you are converting rare catastrophic failures into frequent small visible ones, and it is only an improvement if the organisation is prepared to act on the small ones instead of silencing the alert. Say that out loud when you introduce the standard, and pair it with the operational agreement about who responds and how, or the new signal will be tuned out and the next collapse will look exactly like the last two.