When a Go ingest pipeline's bounded channel stays full, how do you decide between stalling upstream and dropping events?

## Why this is not a coding decision Once the intake is a bounded channel, the code has exactly one open question: what happens when the send would block. Blocking means the producer waits and the stall travels upstream. Giving up means an event ceases to exist. Both are correct implementations; which is right depends on facts the code cannot see — whether the data is replayable, who is harmed by loss, and who is harmed by delay. That is why this is an ownership question. The service owner proposes; the owner of the data can overrule. ## The first question: what does the source do when you stop reading? Stalling is only *free* if the pressure lands somewhere that can hold it. - **A durable upstream** — a log or queue with retention, a producer that retries with its own buffer — absorbs the stall. Consumption falls behind, retention covers the gap, and the visible symptom is lag rather than loss. Stall. - **A live, non-replayable stream** — metrics, traces, click events, anything with no acknowledgement — discards what you do not take, whether or not you admit it. Here "stalling" is really "dropping, invisibly, somewhere else". Shed explicitly instead, so at least the loss is counted. - **A synchronous caller with its own deadline** — an HTTP or RPC client that will time out in two seconds — turns your stall into their failure. Worse, if that caller holds a connection or a pool slot while waiting, your slowness spreads into their capacity and you have manufactured a cascade. Shed fast, and answer with a clear rejection so they can retry or degrade. ## The second question: what is the data worth? A single global policy is almost always wrong because a service rarely carries one kind of event. Classify: - **Must not be lost** — financial events, audit records, anything that has to reconcile. These stall, and if stalling is unacceptable too, they need a durable landing zone: spill to disk or push back to a persistent queue. That costs money and complexity, and it is the honest answer when neither loss nor delay is acceptable. - **Lossy by nature** — telemetry, sampled traces, debug events. These are shed first, and preferably sampled rather than cliff-dropped, so the signal degrades in quality instead of vanishing. - **Everything else** — decide deliberately and write it down, because the default in an incident will be whatever the code happens to do. A good design sheds in that order automatically, so the first thing lost under pressure is the cheapest thing. ## The third question: what do you page on? Depth pinned at capacity is a saturation signal, not necessarily a problem — a burst absorbed and drained is the system working. What deserves a page is sustained saturation, queueing delay past the freshness the consumers of this data were promised, or a shed rate above the agreed budget. Committing to that number is the real deliverable: "we will lose at most X% of class-B events over an hour, and you will be told when we do" is a statement a data owner can accept or reject. "The channel has 1024 slots" is not. ## What makes the decision reversible Everything about this changes under load, so the levers must be operable under load: - Timeouts, queue capacity, sampling rates and the per-class policy in configuration, not constants. - The shed path instrumented from the first line of code, with a counter per class, so the cost of the policy is measured rather than assumed. - A load test that actually drives the service past capacity, because the policy is untested until something has been dropped in anger. - A written note of who agreed to what. The uncomfortable version of this incident is the one where the on-call engineer discovers, at 3am, that they are the person choosing which customer's events to discard. ## The trap to avoid The most common failure is a third option that looks like neither: enlarging the buffer. It feels like a compromise — no stalling, no dropping — but it only converts the decision into queueing delay and memory, and it postpones the moment of choice to a worse time, with a bigger backlog of stale events that may no longer be worth processing when they finally come out. Buffer sizing absorbs bursts; it does not resolve a sustained rate gap. If the arrival rate exceeds the drain rate for long enough, something waits or something is lost, and the only question a leader controls is whether that was chosen deliberately in advance or discovered during an outage.