A job that fell a day behind is now draining that backlog at full throughput — what does that do to the consumers of its output?

## A plateau, not a spike The second-order effect of a backlog is the one nobody plans for. A job that is catching up is, by definition, processing faster than records arrive — that gap is the only reason the distance behind shrinks. Everything it writes goes somewhere, so **the systems downstream of the job receive the input rate plus the drain rate**, and they receive it for the whole drain window, not for a moment. Take a job that normally emits six thousand records a second, fell a day behind, and now runs at nine thousand. Its consumers are seeing a fifty per cent increase, sustained for the nineteen hours the drain takes. Sizing that survives a two-minute spike frequently does not survive nineteen hours of it, because retries, queues and buffers that absorb bursts are dimensioned in seconds. Be precise about the word downstream here: it means the consumers of this job's output, not the steps earlier in the job's own graph. ## What it breaks - **A destination with a write limit throttles.** The important consequence is circular: once throttled, the destination's accept rate becomes the job's processed rate, so the drain rate collapses to `accept rate − input rate` and the recovery stretches out. Recovery time is now set by a system nobody was looking at. - **The next job in the chain inherits it.** A consumer sized for six thousand a second and fed nine thousand builds its own unplanned backlog, and the incident walks one hop downstream at a time. - **Alarms keyed to volume fire.** Anything watching records per minute, rows written per hour or cost per hour reports an anomaly that is real but is a symptom of the recovery, not a new fault. - **Per-request or per-byte billing rises** for the length of the drain, sometimes more visibly than the outage itself did. - **Throttling that is answered with retries wastes the job's own capacity**, so the drain slows further while the cluster looks fully busy. - **Idempotence gets tested.** Elevated write rates and throttling together produce exactly the conditions — timeouts, partial batches, retried writes — under which a destination that is not safe to write twice shows it. ## The cap, and the trade it makes The deliberate remedy is to limit the job's output rate during the drain, trading recovery time for downstream survival. The arithmetic is the same subtraction as before, with the cap in place of the processed rate: ``` drain time = backlog / (output cap - input rate) ``` Two consequences follow, and both are the point of the question: 1. **The cap must exceed the input rate**, or the denominator is zero or negative and the backlog never clears. A cap set "safely" at the consumer's normal rate guarantees permanent lateness. 2. **The margin above the input rate is the whole recovery budget.** A cap ten per cent above arrivals drains ten times more slowly than one at twice arrivals. Choosing it is an explicit negotiation between how long consumers are stale and how hard they are hit. The alternatives are to raise the destination's capacity for the window, to route the drained output to a side destination and merge later, or to tell consumers to expect elevated volume and let it run — all legitimate, all requiring a decision rather than a discovery. ## What varies with the execution model | Model | How the surge arrives | |---|---| | repeated small finite runs, where an endless input is sliced into small finite jobs run back to back | as oversized slices: each catch-up slice covers far more input than a normal one, so output lands in large lumps rather than evenly | | record-at-a-time, each record moving through the graph as it arrives | as a steadily elevated rate for the whole window, the easiest shape to cap | | the two-phase model, each phase writing its whole output to shared storage before the next reads it | as one run emitting several intervals' worth of output at its end, which is the least gradual of the three | The lumpy cases are worse for a rate-limited destination than the smooth one, because a limit is enforced per second and a lump is not spread. ## Planning for it before it happens The senior answer is that the drain plan exists before the incident: you already know which destination has the lowest write ceiling, so you know what the binding constraint on recovery will be; the output cap is a value you can set without a redeploy; and the people who own the consuming systems are told that a recovery looks like elevated volume, so the second incident is not caused by the response to the first.