convertAndSendToUser works on a single instance but silently fails in a multi-node cluster. Why, and how do you fix it?

**Root cause — the SimpUserRegistry is local:** When a user connects, Spring records their sessions in a `SimpUserRegistry`. The default implementation is **in-memory and per-JVM** — it only knows about WebSocket sessions terminated on *this* instance. `convertAndSendToUser(user, dest, payload)` asks the registry "which sessions does `user` have?", rewrites the destination per session, and sends. On the node where the user *isn't* connected, the registry returns nothing, so the send is a **silent no-op** — no error, no delivery. Fine on one node; broken the moment you scale horizontally behind a load balancer. **Why the simple broker can't help:** `registry.enableSimpleBroker("/topic","/queue")` is an in-memory broker living inside each JVM. Two app instances have two independent brokers that don't share subscriptions or messages. So even `/topic` broadcasts don't cross nodes with the simple broker, let alone user destinations. **Fix — external broker relay + two broadcasts:** ```java @Override public void configureMessageBroker(MessageBrokerRegistry registry) { registry.enableStompBrokerRelay("/topic", "/queue") .setRelayHost("rabbitmq-host").setRelayPort(61613) .setUserRegistryBroadcast("/topic/simp-user-registry") .setUserDestinationBroadcast("/queue/unresolved-user-destination"); registry.setApplicationDestinationPrefixes("/app"); } ``` 1. **`enableStompBrokerRelay(...)`** — instead of an in-JVM broker, each app node connects (via STOMP) to a shared external broker (RabbitMQ with the STOMP plugin, ActiveMQ, etc.). All nodes now share `/topic` and `/queue` state through that broker. This alone fixes normal broadcasts across the cluster. 2. **`setUserRegistryBroadcast(dest)`** — nodes periodically **broadcast their user registry** over the given broker topic, and each node merges the others' data into a `MultiServerUserRegistry`. Now node A *knows* user X is connected (somewhere), even if not locally. Used for things like presence/`SimpUserRegistry.getUser(...)` queries cluster-wide. 3. **`setUserDestinationBroadcast(dest)`** — when a node receives a message for a user destination it **can't resolve locally**, it forwards (broadcasts) that unresolved message over this broker queue. Every node listens; the node where the user *is* connected resolves and delivers it. This is what actually makes `convertAndSendToUser` reach a user on another node. You need **both** broadcasts: the registry broadcast for cluster-wide *knowledge* of users, and the destination broadcast for *delivery* of unresolved user messages. **Operational considerations (principal-level):** - **Broker becomes critical infra:** you now depend on RabbitMQ/ActiveMQ availability, credentials, TLS, and the broker's own clustering/HA. The relay maintains a system TCP connection (`setSystemLogin`/`setSystemPasscode`, heartbeats). - **Sticky sessions still apply** at the HTTP/SockJS layer: a single logical WebSocket/SockJS session's transport requests must hit the same node; the broadcasts solve *cross-user* routing, not moving a live session between nodes. - **Scaling the fan-out:** registry broadcast traffic grows with node count (each node advertises its users to all); for very large clusters this chatter and the unresolved-destination broadcast add overhead. Alternatives include partitioning users by node (consistent hashing at the LB) so cross-node sends are rare, or using the broker's own routing. - **Ordering/at-most-once:** unresolved-destination broadcast is best-effort; if a user reconnects to a different node mid-flight, a message may miss. Design idempotent/replayable delivery for anything critical (persist notifications, let the client fetch on connect). - **Security:** the broker relay must be authenticated and network-isolated; destinations used for broadcasts (`/topic/simp-user-registry`, `/queue/unresolved-user-destination`) are internal — ensure your message-security rules and broker ACLs don't expose them to clients. **Diagnosis tip:** the symptom is 'works locally, nothing in prod behind a LB, no exception.' Check whether you're on `enableSimpleBroker` (single-node) and whether the two broadcast destinations are configured on the relay. `SimpUserRegistry` being a plain in-memory instance rather than a `MultiServerUserRegistry` confirms the gap.