An adversary fires 300 near-identical alerts so your response platform hits its API rate limit — how do you stop real containment being dropped?

## The shape of the failure A response platform's connectors talk to the same APIs as everything else: the EDR, the identity provider, the artifact repository, the cloud control plane. Those APIs enforce per-tenant rate limits and answer an over-quota call with HTTP 429 (Too Many Requests), usually carrying a `Retry-After` header that says when to try again. A 429 is a refusal — the request was not performed and nothing was queued at the target. Most of a SOC's API budget is spent on *enrichment*, not on response: for every alert, a playbook pulls the asset record, the user's recent sign-ins, the process tree, a handful of reputation lookups. That is dozens of calls per alert against a budget that is shared with the small number of calls that actually change the world. When alert volume spikes, enrichment consumes the quota first and containment — which happens later in the run — collides with the limit. ## Why the flood is cheap for the adversary and expensive for you An adversary does not need to compromise anything to trigger this. Repeatedly touching something a noisy rule watches — a monitored registry path, a flagged binary name, a login pattern that trips an identity detection — produces alerts at essentially zero cost, from a position that does not matter, on hosts that do not matter. Each one launches a playbook that spends real API budget. Three hundred of them are minutes of work. The payoff is asymmetric: the one case that matters arrives inside the flood, its containment action gets a 429, and the case proceeds as if containment happened. Even without exhausting the quota, the flood buries the real case in a queue and buys dwell time. Treat a sudden burst of low-value, near-identical alerts as a possible diversion rather than as a nuisance to be silenced. ## Designing the response path so it survives **Do not launch one run per alert.** Collapse alerts that share an entity — the same host, the same account, the same artifact — into one case, so three hundred alerts drive one investigation and one set of actions. This alone removes most of the load. **Split the quota by consequence.** Route containment and eradication actions through a prioritised lane with a reserved share of the budget, and let enrichment run in a lower lane that yields. Enrichment being late costs you context; containment being dropped costs you the case. **Cap concurrency per playbook class.** A bounded worker pool with a queue is better than unlimited parallelism: unbounded runs sprint into the rate limit together, and every one of them fails. **Retry deliberately, not reflexively.** Honour `Retry-After` and use bounded backoff. Immediate retries deepen the limit for everyone, including the action you care about. Bound the total wait, and when it is exhausted, escalate rather than give up quietly. **Fail loud on the actions that matter.** A rate-limited containment action must set the case to a blocked state and page a human with the specific action that could not be performed. A responder who knows the automation could not isolate the host will do it by hand in a minute; a responder who sees a green run will do nothing. **Keep a path that does not share the quota.** The manual console route, or a separate integration for emergency actions, so an exhausted automated path never means no path. ## Detecting it while it is happening Monitor the automation platform's own execution record as a telemetry source in its own right: the rate of 429 responses per connector, the count of playbook runs queued or dropped, and the time from case creation to first containment action succeeding. A spike in 429s during an incident is an operational alarm and possibly a security signal. Both readings deserve a look, and you cannot distinguish them from the alert stream alone. ## What to do afterwards Two pieces of follow-up work: 1. **Re-verify the affected cases.** Enumerate the actions that returned 429 during the window, check each target's audit trail for whether the action ever landed, and perform the ones that did not. Any case containing a dropped containment action was not contained when it was said to be. 2. **Hunt the window.** Ask what else was happening while the SOC's attention and API budget were consumed. Look at the surfaces the flood did not touch — identity sign-in logs, cloud control-plane audit entries, egress flow records — over exactly that interval. A diversion is only worth running if something is happening behind it, and the flood gives you an unusually precise time box in which to look. ## The distinction to hold onto A broken detection producing three hundred alerts and an adversary generating three hundred alerts look identical for the first ten minutes. The difference is found by evidence — a recent rule change, an unrelated deployment, the spread of affected hosts — not by assumption. Until that evidence exists, the safer default is to assume the volume is intentional and keep the response path protected.