How would you decide how many minutes-long, held-open streaming responses one instance should carry, and how would you shed them safely?

An endpoint that holds a response open for minutes is a different kind of object from a request that finishes in milliseconds. Its cost is not measured in requests per second but in **occupancy**: how many are resident at once, what each one pins, and for how long. Capacity for these is a design decision, and the shedding plan is part of the design rather than an afterthought. ## Start by measuring what one stream costs Do not reason from first principles about this; hold N streams open on one instance and measure. The per-stream cost is the sum of: - **A connection**, with its kernel buffers and its slot against any connection or descriptor limit on the box and on every intermediary in front of it. - **Framework state** — the per-connection response buffer, and whatever queue the slow-reader policy allows to build. - **Whatever the execution model pins.** This varies enormously: models that dedicate a worker per in-flight request pay a large fixed cost per open stream, while models that suspend a cheap continuation between writes pay very little. The same code can differ by orders of magnitude in ceiling depending on which one it runs on. - **Whatever the handler holds** — a cursor, a pooled connection, a subscription, a cached snapshot. This is the cost people forget, and it is frequently the binding one, because pools are small. Then find which limit binds first. It is usually the pool or the worker count, not memory, and the answer changes the design rather than the sizing. ## The ceiling is only half the answer Three forces make a static number insufficient: 1. **Intermediaries close idle connections.** A path with proxies and load balancers has its own idle timeouts, and a stream that produces nothing for a while is indistinguishable from a dead one. Keeping the path alive means writing something periodically, and those periodic writes are themselves load — at a high stream count, a keep-alive interval is a throughput decision, not a detail. 2. **Long streams pin old instances.** A deployment cannot drain what does not end. Without a bound, a rollout either waits for streams that may never finish or cuts them mid-flight, which is the same outcome with worse timing. 3. **Streams do not rebalance.** Once established, a stream stays on the instance that accepted it for its whole life, so scaling out only helps *new* streams. An instance that filled up during an incident stays full. All three point at the same countermeasure: **bound the lifetime.** Give every stream a maximum duration after which the server closes it cleanly and the client establishes a new one. That turns an unbounded occupancy problem into a rolling one — rollouts drain, load rebalances, and leaks are bounded by the lifetime. It is cheap when the client can resume from a position it already knows, and that resumability is worth designing in for exactly this reason. ## Shedding, in the order it should be tried | Mechanism | What it protects | What it costs | |---|---|---| | Admission cap per instance | The ceiling itself, deterministically | Some clients are refused and must retry elsewhere | | Bounded per-stream queue | Memory, against slow consumers | Skipped records the contract must disclose | | Maximum stream lifetime | Drainability and rebalancing | Periodic re-establishment by clients | | Closing the least valuable streams | Headroom under pressure | Requires a priority notion the system must define | Refusing a stream at admission with a clear, retryable status is far better than accepting it and degrading everyone, because the refusal is visible and the degradation is not. The only rule that matters in ordering these: shed at the door before you shed in the middle. ## The design questions to raise - **Does this need a held-open stream at all?** If updates are infrequent and the consumer tolerates delay, a cheaper interaction may carry the same information at a fraction of the occupancy. Reserve held-open streams for cases where latency genuinely is the feature. - **One producer feeding many streams, or one query per stream?** Fan-out from a single source keeps per-stream work small; per-stream querying multiplies backend load by subscriber count and is the usual cause of the second outage. - **Should these live on a separate deployment?** Long-lived streams and fast request/response traffic have opposite operational profiles — one wants long drains and high connection counts, the other wants quick rollouts. Isolating them stops a stream surge from starving ordinary traffic, at the price of another thing to run. - **What is promised to the consumer?** Maximum lifetime, whether records may be skipped, and what a clean end looks like all belong in the interface contract, because every shedding mechanism above is visible to clients. ## The summary a lead should be able to give Measure the per-stream cost on the real execution model, find the limit that binds first, cap admission below it, bound every stream's lifetime so the fleet stays drainable and rebalanceable, bound the per-stream queue so one slow consumer cannot take the instance, and write all of that into the contract. A number without those mechanisms is a guess that will be wrong the first time traffic is unusual.