skip to content

For a messaging gateway fleet, would you place each user's connection on a hash-determined node or on any node tracked by a registry, and why?

level: principalimportance: nice to knowfreq 30%

answer

  1. computed versus looked up
  2. what the router must read
  3. cost of adding one node
  4. new nodes and old connections
  5. who owns rebalancing

basics

~20 s

Hash placement removes the registry lookup but needs user-aware routing and moves users when the node set changes; any-node placement plus a registry keeps routing simple and elastic, at the cost of a lookup and slow rebalancing.

solid answer

~50 s

With **hash placement**, the router reads the user ID from the upgrade request and uses consistent hashing to pick the node, so a backend computes where a user lives without a lookup and all of a user's devices land together. The price: routing must parse and authenticate at L7, hot users skew nodes, and every scale event moves about 1/N of users; adding a 21st node to 20 moves roughly 476,000 of 10 million connections, which must be drained gradually. With **any-node placement**, a plain L4 balancer spreads connections, nodes join and leave without moving anyone, and a registry answers the lookup, but that registry is a critical dependency and new nodes fill only as connections churn, so hot nodes must shed load deliberately. I default to any-node plus registry for elasticity, and choose hashing when lookup cost or registry availability is the harder constraint.

go deeper

for a junior

Recall the two ways to know where a user is connected: compute it from the user ID, or look it up in a shared table.

for a middle

Explain consistent hashing and why it moves only about 1/N of users, and why an L4 balancer cannot see the user ID.

for a senior

Quantify the operational costs: users moved per scale event, hours of skew after scale-out, and how to rebalance at a controlled rate.

for a principal

Pick from the system's constraints, such as elasticity, lookup cost, traffic skew and what the organisation already operates, and say when a hybrid is worth it.

## The decision Every real-time messaging system has to answer *"which gateway node holds this user?"* Two families of answers exist. **Deterministic placement** makes the node computable from the user ID. **Any-node placement** lets a connection land anywhere and records where it landed in a shared registry. Both work at large scale; the choice depends on which costs a given system tolerates better. ## Option A: deterministic placement The router in front of the gateways reads the user ID from the connection's upgrade request, typically from an authenticated token, and applies **consistent hashing** to pick the node. Consistent hashing places nodes and keys on a hash ring so that adding or removing a node reassigns only about 1/N of the keys, instead of nearly all of them as `hash mod N` would. Strengths: - a backend computes the node from the user ID, with **no registry lookup** on the delivery path; - all devices of one user land on the same node, which simplifies per-user fanout; - there is no shared routing store to keep available. Costs: - the router must work at **L7**, parsing and validating each upgrade request, which is heavier than forwarding connections; - **changing the node set moves users.** Growing from 20 to 21 nodes reassigns about 1/21 of users: with 10 million connections, roughly **476,000** must be moved, gradually, or messages reach the wrong node; - **hot users**, such as accounts with many devices or heavy traffic, cannot be spread, so a node's load depends on who happens to hash there; - during a ring change, senders and the router must agree on the ring version, or a message is routed by the old ring. ## Option B: any-node placement with a registry A plain **L4** balancer spreads connections without looking inside them. The gateway that accepts a connection writes a user-to-gateway entry to a shared registry, and delivery reads it. Strengths: - the routing layer is simple and cheap; - nodes join and leave **without moving anyone**; - per-node load follows connection counts, not where users hash. Costs: - every delivery pays a **registry lookup**, and the registry becomes a critical dependency that needs replication and a way for gateways to re-register after a failover; - **scale-out fills slowly**, because new nodes receive only newly opened connections. ## The scale-out skew, in numbers Assume 20 nodes holding 500,000 connections each, and the fleet adds 5 nodes. The balanced target is 10,000,000 / 25 = **400,000** per node, so the new nodes need 5 x 400,000 = **2,000,000** connections between them. If 5% of connections churn per hour, 500,000 connections open anew each hour; even if every one lands on a new node, balance takes at least 2,000,000 / 500,000 = **4 hours**. If the scale-out was a response to overload, that is far too slow. The remedy is **active rebalancing**: overloaded nodes ask a small, randomized fraction of their clients to reconnect, at a capped rate, until they reach target. With a balancer that favours less-loaded nodes, those reconnects land mostly on the emptier ones. ## Side by side | Dimension | Deterministic placement | Any-node plus registry | |---|---|---| | Delivery lookup | computed, no store read | registry read per delivery | | Routing layer | L7, user-aware | L4, connection-level | | Scale event | moves about 1/N of users | moves nobody; new nodes fill slowly | | Hot users | pinned to their hashed node | spread by connection count | | Shared dependency | agreed ring configuration | registry store | | Rebalancing trigger | ring changes | operator or automation | ## How to decide 1. **How elastic is the fleet?** Frequent autoscaling favours any-node placement, because adding capacity never forces moves. 2. **How costly is a lookup?** If delivery volume is enormous and latency-critical, removing the lookup is worth more. 3. **How skewed is per-user traffic?** A heavy tail argues against hashing. 4. **What can the team operate?** A highly available registry and a user-aware L7 router are both real investments; favour the one the organisation already runs well. Hybrids exist. A system can hash users to a **group** of nodes for coarse locality and keep a registry within each group, or keep deterministic placement and consult a small registry only for users who are mid-move during a ring change. A strong answer names the costs of both options and picks from the system's constraints rather than declaring one universally right.

  • How do you rebalance an any-node fleet after scale-out without causing a storm?
    Have overloaded nodes send a reconnect hint to a small, randomized fraction of their connections at a capped rate, and stop once they are near the fleet average. A balancer that favours less-loaded nodes steers those reconnects toward the new ones; even an even spread shifts load onto them over time. Admission control on every node remains the backstop if the rate is set too high.
  • Can the two placement strategies be combined?
    Yes. One hybrid hashes users to a group of nodes for coarse locality and keeps a registry within each group, so lookups stay small and scaling a group moves nobody. Another keeps hash placement but consults a small registry only for users who are mid-move during a ring change, so messages still reach them.

saying these in an interview costs you the question

  • With consistent hashing, adding a gateway node moves no connections.
  • Any-node placement rebalances itself as soon as nodes are added.
  • A plain L4 balancer can route by the user ID inside the upgrade request.
  • Any-node placement works without a registry or other lookup.
  • One placement strategy is right for every messaging system.