How would you set disruption-cap policy across a shared cluster where maintenance windows keep expiring with drains still blocked?

## What the policy is arbitrating A disruption cap is a rule one team writes and another team pays for. The workload owner writes 'keep this many of my copies available', and the person who has to empty a host tonight to flash firmware or swap a failing disk is the one whose window expires. Neither side is wrong: unpatched hosts are a real risk with a clock on it, and an ingester dropping below its serving capacity at 02:00 is a real incident. A policy that does not say who yields yields to whoever waits longest - which, in the situation described, is the maintenance window. ## Four rules worth writing down 1. **A cap has to be affordable.** A workload asking to keep N copies available must declare more than N copies. A floor equal to the declared count is arithmetically unsatisfiable - no drain of any host it runs on can ever proceed - and that should be rejected when the cap is declared, not discovered during a window. 2. **Prefer a cap expressed against the copy count.** An absolute 'at most one down' means something very different at ten copies than at two, and a workload whose count moves through the day changes the strictness of its own cap with nobody editing it. Where the platform supports a proportional form, use it, and check what it does at your smallest copy count, because **platforms differ in how they round** and the small end is where drains block. 3. **Give undeclared workloads a default.** Most workloads never declare a cap, so the default decides whether the fleet is drainable at all. A loose default plus an exception process works; a strict default applied silently to everything does not, and you will find out months later. 4. **A blocked drain alerts the owner, with a deadline.** The operator cannot fix a cap, add copies, or decide that a workload may take a gap. Route the alarm to the team that can, state how long the drain will wait, and write down in advance what happens when that deadline passes. ## Absolute against proportional | | Absolute cap | Proportional cap | |---|---|---| | At ten copies | mild, roughly a tenth of headroom | tracks the count | | At two copies | strict, frequently blocking | tracks the count, subject to rounding | | When the count changes | strictness changes silently | meaning stays stable | | Easy to reason about | yes | needs a rounding check | ## The escape hatch, named in advance Some platforms let an operator force a removal past a cap and some do not. Where the capability exists, the policy question is not whether to use it but **who may**, and the answer should never be 'whoever holds the pager at the time'. Put the breach decision with the owner who set the cap or with an on-call lead, require that each use is recorded, and treat every use as evidence that a cap or a copy count needs changing. A cap that is routinely and silently breached is worse than no cap: it still blocks honest automation while protecting nothing. ## What you are actually trading - **Patch latency against availability.** Every hour a drain waits is an hour a host runs old firmware, and fleets that cannot be emptied accumulate that debt across every host at once rather than one at a time. - **Automation against attendance.** Loose caps let unattended maintenance run overnight; strict caps demand a human per workload per window, which does not scale past a few dozen services. - **Whose incident it is.** A loose cap turns a window into a small availability dip the owner did not schedule. A strict one turns it into an expired window the operator did not schedule. Pick deliberately, per workload, and say so out loud. ## The failure to design against The outcome to avoid is not a blocked drain; it is a **fleet that quietly stopped being drainable**. It arrives as a handful of workloads with unsatisfiable caps nobody reviews, an operator who has learned to force past them as routine, and a set of hosts that have not been emptied in months. Every rule above exists to make that state visible while it is still three workloads, rather than after it has become the way the cluster works.