A team wants one shared stream writable at both sites — why do operators propose one stream per site instead, and what does that cost readers?

## What the request actually asks for 'One shared stream, writable at both sites' asks for a single named object with **two independent writers** and one agreed content. Everything that goes wrong in this arrangement descends from those two writers: - Two copiers facing each other echo a record back and forth without bound unless every hop carries an origin stamp and honours it. - Two sites sequencing the same stream produce two orderings, and no after-the-fact merge recovers a true order across them, because none was ever recorded. - Readers at the two sites see different interleavings of the same records, so state folded from the stream differs by site. ## What one stream per site changes A **per-site stream** is written only at the site that owns it and read everywhere. Each site keeps producing, and each site still ends up holding all the records. The difference is that no stream ever has more than one writer. | Property | One shared stream, written at both sites | One stream per site | |---|---|---| | Loop prevention | mandatory: every hop must stamp and filter | unnecessary: nothing copies into a stream another site writes | | Ordering | undefined across the two sites | each stream carries its single writer's order | | Duplicates | the echo, plus copier restarts | copier restarts only | | Reader work | one subscription, one apparent sequence | one subscription per site, combined explicitly | | Adding a site | another pair of stamped hops and a new source of unordered writes | one more stream, read wherever the others are | | Where combining happens | inside the stream, implicitly, on no defensible rule | inside the reader, visibly, on a rule someone chose | The last row is the whole argument. The merging work does not disappear; it moves from a place where nobody can see or justify it to a place where a person had to write it down. ## The cost, stated honestly - **Fan-in moves to the reader.** A consumer that read one name now reads several, and adding a site changes what every consumer reads. Where the reader is one process treating the streams as a set, that is trivial; where it maintains per-entity state, the combination rule is now its explicit problem. - **Cross-site aggregates get harder before they get better.** Any 'total across all sites' answer now depends on the reader holding every stream and on the slowest hop feeding it. - **The stream count multiplies.** One logical subject becomes one stream per site, which matters wherever a platform charges per stream, caps how many exist, or spends fixed overhead on each. - **Hops grow with the square of the sites.** If every site must hold every other's streams, an estate of *n* sites wants n(n−1) directed hops. That, not ordering, is usually what caps the shape in practice; beyond two or three sites, estates route through a hub or abandon the idea that every site holds everything. ## When a shared writable stream is still defensible It is defensible where nothing downstream depends on order across sites and duplicates are harmless — an append-only measurement or audit feed folded into counts, or a stream whose single consumer treats it as a set. Even then, the stamping discipline on every hop becomes a standing obligation someone must maintain forever, including on the hop added at two in the morning during an incident by whoever is on call. ## What a lead is really being asked to decide Most requests for writes at both sites are one of two other requests wearing a costume: 1. **'Writers must keep working when their nearest site is unreachable.'** That is a routing question — where a writer goes when its site is gone — and it is answered by the switch onto a standby, not by dual acceptance. 2. **'Readers want the data locally.'** That is answered by copying a single-writer stream outward, which is exactly the per-site arrangement. Separating those two usually ends the argument, and naming which one was actually asked for is the move expected at this level. The remaining honest questions are whether the organisation will ever rehearse the failure the design exists for, and whether it will keep the discipline the design needs on every hop anyone adds later. ## Where platforms differ - On a queue-shaped broker, one queue per site drained by competing consumers is close to the natural shape already, so the per-site arrangement costs little conceptually. - On a platform that splits a stream into parts, per-site streams multiply the parallelism accounting: each stream carries its own part count, and the reader side has to be sized against the sum rather than one number. - On a rented cluster the ceiling may not be yours to raise. Stream counts, hop counts and cross-site bandwidth can each be capped by the offering, which changes the arithmetic before any design argument begins.