Which catch-up levers should an estate pre-authorise for whoever is on call, and which should still require the stream's owner?

Every drain is run under time pressure by someone who did not design the stream. The purpose of pre-authorisation is to move the decisions that need thought out of that moment and into a calm one, and the useful dividing line is not how dangerous a lever feels but whether pulling it changes what the system computes. ## Two tiers, drawn on correctness 1. **Reversible and result-neutral — pre-authorised.** Adding reader instances up to the parallelism ceiling, and raising how many records a reader takes per request within a stated bound. These change throughput, spend and per-record latency. They can be undone. No downstream result differs because they were pulled. An operator asking permission for these is wasting the only resource the incident is short of. 2. **Result-changing — the stream owner's call, made in advance.** Waiving in-order handling to unlock parallelism, and diverting records that block progress to a side destination. Whether either is acceptable depends entirely on what the records mean and what the handler does with them, which the on-call engineer usually does not know. The answer can and should be written down long before it is needed. A third category exists and is not a drain lever at all: anything that moves the reader group's recorded read position changes which records are ever handled, and belongs with the stream's owner as a decision in its own right. ## What the artefact looks like A per-stream statement, short enough to be read during an incident and stored where the alert points, not in a design document nobody opens: - **Maximum members** it is worth starting, and whether that is bounded by the stream's shape or by a shared downstream system. - **Batch bound** — the largest records-per-request value that has been tested against the progress deadline, and whether it must be put back afterwards. - **Ordering waiver** — permitted, permitted with conditions, or not permitted, with one sentence of reasoning that survives the person who wrote it. - **The owner to call** when the arithmetic says the backlog cannot be drained intact. - **Expected drain rate per member**, so an on-call engineer can compute the surplus without instrumenting the system first. | Lever | Who decides | When | |---|---|---| | More members to the ceiling | On call | During the incident | | Larger batches within bound | On call | During the incident | | Waiving in-order handling | Stream owner | Recorded in advance | | Diverting blocked records | Stream owner | Recorded in advance | | Moving the recorded read position | Stream owner | Its own decision, never incidental | ## Why this is a principal's problem rather than a runbook's Three reasons the list cannot simply be delegated to each team. - **Consistency across an estate.** If one stream permits an ordering waiver and the next silently does not, on-call engineers rotating across both will eventually apply the wrong default. The shape of the statement has to be the same everywhere even though its contents differ. - **It expires.** A waiver reasoned about a handler that has since been rewritten is worse than no waiver, because it carries authority it no longer deserves. The statement belongs to the stream's ownership record and is reviewed when ownership changes. - **Authorisation without capacity is theatre.** Permission to add members buys nothing if there is no headroom to add them into and no path to purchase it at three in the morning. Deciding whether recovery headroom is a standing cost or an on-demand purchase is a budget decision, and it is the one that actually determines whether the surplus in the arithmetic can be reached. ## The failure this prevents The characteristic bad outcome is not an operator who hesitates; it is an operator who does not. Under pressure, with a gap climbing and a retained window somewhere ahead, the tempting move is the one that unlocks the most parallelism fastest — and on many streams that move quietly changes results in ways nobody notices for weeks. Pre-authorisation is what makes the fast levers genuinely fast and the dangerous one genuinely a decision. It also has a side benefit: writing the statement forces someone to work out whether the stream's handler actually depends on order, and that question has a way of surfacing assumptions that were never true. ## What stays out of it This is a per-stream operating agreement about what may be traded for throughput. How far a gap may grow before anyone is woken, and how much unavailability the service is allowed to accumulate, are different documents owned elsewhere. Mixing them produces a page nobody reads.