When deploying a WebSocket gateway fleet holding millions of connections, how do you drain nodes without triggering a reconnect storm?

## Why deploying a gateway is different Deploying a stateless service is routine: stop routing requests to an old instance, wait a few seconds for in-flight work, stop it. A **gateway node** in a real-time messaging tier may hold 500,000 connections, some of them open for days. Stopping it disconnects every one of those clients, and each client reacts the same way, by reconnecting to the remaining nodes. ## Anatomy of a reconnect storm A **reconnect storm**, a form of **thundering herd**, happens when many clients that lost their connections at the same moment reconnect at the same moment. Illustrative arithmetic: - One node with 500,000 connections restarted abruptly: if its clients all reconnect within about a second, the fleet sees roughly **500,000 connection attempts per second**. - The same clients spread evenly over 60 seconds: 500,000 / 60, about **8,333 per second**. - A whole fleet of 10 million connections recovering from a regional outage is twenty times the single-node burst. Each attempt costs a TCP connect, a TLS handshake, an HTTP upgrade, authentication and a registry write. TLS handshakes are CPU-heavy, so healthy nodes saturate, handshakes time out, clients retry, and the retries add more load. That feedback loop can keep the tier down long after the original cause is gone. ## A drain sequence 1. **Stop new arrivals.** Fail the node's readiness check so the load balancer sends it no new connections. Existing connections are untouched. 2. **Let natural churn help.** For a short period, clients that disconnect on their own simply land elsewhere when they return. 3. **Close the rest at a controlled rate.** Send each client an application-level "please reconnect" message carrying a randomized delay, then close. Draining 500,000 connections over 10 minutes is 500,000 / 600, about **833 closes per second**. 4. **Force-close at a deadline.** Some connections would otherwise stay open for days, so the drain has a maximum window. 5. **Roll in batches.** Drain only as many nodes at once as spare capacity can absorb. With 20 nodes, batches of 2 and a 10-minute drain, the rollout takes 10 batches x 10 minutes = **100 minutes**. Because this is slow, the gateway should stay **thin**: logic that changes weekly belongs in stateless services behind it, so gateway deploys stay rare. ## Client-side discipline Clients must not retry in lockstep: - use **exponential backoff** with a cap, so repeated failures slow down; - add **full jitter**, sleeping a random time between zero and the current cap, so clients dropped together spread out; - honour a delay the server supplies, whether in a reconnect message or a `Retry-After` header; - reset the backoff only after a connection has stayed up for a while, so a flapping server does not send everyone back to zero. ```pseudocode attempt = 0 loop: result = connect() if result is connected: opened_at = now() wait until the connection closes if now() - opened_at > STABLE_PERIOD: attempt = 0 else if result is HTTP 503 with Retry-After: sleep(Retry-After + random(0, JITTER)) attempt = attempt + 1 continue cap = min(MAX_DELAY, BASE_DELAY * 2 ^ attempt) sleep(random(0, cap)) attempt = attempt + 1 ``` Note that after any close, including a planned drain, the loop still sleeps a random time before reconnecting. ## Server-side admission control Not every client is well-behaved: old app versions and buggy retry loops exist. Each node therefore protects itself: | Control | What it does | |---|---| | Accept-rate limit at the listener | caps new TCP connections per second before any TLS work is done | | Upgrade token bucket | rejects excess upgrade requests with `503 Service Unavailable` and `Retry-After`, skipping authentication and registry writes | | Connection ceiling | refuses new connections once the node reaches its planning target | | Randomized `Retry-After` values | spread rejected clients over a window instead of one instant | A **token bucket** refills at a fixed rate and each new connection spends a token, so a node admits a steady flow and turns away the excess cheaply instead of half-completing thousands of handshakes. ## Choosing the drain speed A faster drain finishes deploys sooner but concentrates reconnects; a slower one is gentle but stretches deploys into hours. Set the rate from how many handshakes per second the rest of the fleet can absorb with headroom, and rely on admission control to catch whatever the estimate missed.