Cluster Shape & Capacity

Broker storage is dominated by ordered appends, and most reads arrive moments later asking for the same bytes in the same order, so the volume is asked for steady sequential bytes per second rather than fast random seeks.

Node-level failures only: a disk, a process, one machine. The rack is a single power and network domain, so one rack event takes all three copies at once — three copies against one failure class, one copy against another.

A cluster splits two jobs. Record-serving nodes accept writes and serve reads for the partitions or queues they own. The metadata role holds membership, stream and partition metadata and configuration, and admits every structural change.

On platforms that split a stream into partitions, the count is the parallelism ceiling: each partition is normally served to one reader at a time, so extra readers idle. A shared queue has no such count; competing readers simply take more work.

An even split leaves each survivor holding exactly half, and half is never a majority. Losing either domain therefore leaves no majority of the copy set or of the membership holding cluster metadata. A third domain makes one survivor decisive, even if it only breaks ties.

Every change to cluster state must be agreed by a majority of that membership, so each extra member raises the number that must agree and the work per change without adding proportional failure tolerance. Odd sizes avoid paying for a member that tolerates nothing extra.

The change is asymmetric. Raising the partition count on a stream is a supported, routine operation; lowering it generally is not offered at all, so coming down means creating a second stream at the smaller count and moving writers and readers across.

Rate times average record size gives the write side: 20,000 x 2 KB is about 40 MB per second in. Each independent reader set that reads every record adds the same again, so four make roughly 160 MB per second out.

Structural change stops: no new streams, no configuration change, no partition moves, no recorded change of ownership. On many platforms traffic to partitions whose ownership is already settled keeps flowing, though some designs fail closed instead. The cluster is frozen in its current shape.

The cluster answers the first connection with its own view of itself: the address each member announces, and which member currently serves which work. Every later connection targets those announced addresses, not the address the client originally dialled.

Whether that record is still resident in memory somewhere: the operating system's file cache, the broker process's own memory, or neither. On designs that cache little, and for any reader that has fallen far behind, every such read goes to the volume.

Each partition carries fixed overhead on every node holding a copy of it, multiplied by the copy count and by every stream in the estate. A very large total also lengthens recovery and enlarges the metadata the coordination membership must agree on.

The busiest interval, not the daily mean. A peak-to-average ratio of five means the machines must carry 40,000 records per second, while the volume size still comes from the sustained rate accumulated over the retention span.

The entry address is reachable from the client but the per-node addresses the cluster handed back are not, so every connection after the first goes nowhere. The fix is a second announced view whose addresses are reachable from where that client sits.

Suspect how the volume's speed was provisioned: a throughput allowance bought separately from size, or a volume shared with another workload. The append path is being made to wait, and waiting is not an error, a fault or a space alarm.

The layout was correct when built and drifted afterwards. Nodes replaced one at a time rejoined with a missing or wrong domain label, so the cluster placed copies as though every node were independent. Counts stayed full; the spread behind them did not.

The slowest resource every reader shares while handling a record - a database, a pool, an external service's allowed call rate. The queue imposes no count, so the useful number of competing readers is the one that saturates that resource and no more.

Because a write is not finished when it is accepted. Where one node takes the write and sends it to the other two, internal traffic is about twice client ingress, and every stored copy also writes the full record bytes to its own volume.

Neither wins outright. Local volumes give the highest and least variable throughput but die with the node; network-attached volumes survive the node and can be reattached, paying a network hop in the write path and a throughput allowance you must buy.

Usually: every cluster spans three independent domains, with neither the copies of a partition nor the membership holding cluster metadata leaving a majority in any one. It forecloses single-rack clusters, two-domain locations without a tie-breaker, and any domain far enough away to sit in the write path.

An internal membership means one system to size, patch and reason about, but coordination competes with record traffic unless separated. A separate coordination service isolates cluster state at the cost of a second system with its own majority, upgrades and on-call. A hidden one removes both jobs and all visibility.

Set a small default for the long tail of low-rate streams and publish one or two higher tiers a team opts into with a stated reader-parallelism need. A generous single default is multiplied by every stream anyone ever creates.

Three things reliably arrive on top of measured traffic: a reader replaying from the start, a writer retry surge re-sending records already accepted, and a dead node's share landing on its surviving peers. The design target is a fraction of what the machines can do, not all of it.

The rule is that every client population gets a view whose addresses are usable from where it sits, and that the number of populations stays deliberately small. Each extra view is one more address per member to keep correct for the cluster's life.