Broker & streaming operations

The entry address only has to reach any one member, because the cluster then describes itself and hands back the per-node addresses later connections use. A hand-written list of every node duplicates membership the cluster already knows, and goes stale.

Broker storage is dominated by ordered appends, and most reads arrive moments later asking for the same bytes in the same order, so the volume is asked for steady sequential bytes per second rather than fast random seeks.

Node-level failures only: a disk, a process, one machine. The rack is a single power and network domain, so one rack event takes all three copies at once — three copies against one failure class, one copy against another.

A cluster splits two jobs. Record-serving nodes accept writes and serve reads for the partitions or queues they own. The metadata role holds membership, stream and partition metadata and configuration, and admits every structural change.

On platforms that split a stream into partitions, the count is the parallelism ceiling: each partition is normally served to one reader at a time, so extra readers idle. A shared queue has no such count; competing readers simply take more work.

A full data volume is a refusal, not a slowdown: the node stops accepting writes, and it stops accepting them for every stream whose data lives on that volume, not only the one that filled it. Reads usually keep serving.

Whichever binds first. Both bounds are live at the same time, and the oldest records become removable as soon as either the age is reached or the bytes are reached, so a busy stream hits the byte ceiling long before seven days.

Keyed retention promises only that the latest value for every key stays readable. It never states when a superseded value is removed, so it is a guarantee about what survives, not a size or deadline lever.

Remote offload moves only closed segments to a remote object store; the segment still being appended to stays on the data volume. History stops being bounded by the local device, while the write path is untouched, so it buys capacity, not speed.

The removal pass discards whole closed segments oldest-first, never individual records, so it cannot be aimed at one subject. A retention floor usually forbids dropping that history early in any case. Erasure therefore has to make records unreadable rather than absent.

A write can be answered with no wait at all, once the leader holds it, once a majority of copies hold it, or once every caught-up copy holds it. Each rung up adds a network round trip of latency and removes one way to lose the record.

The third copy adds a complete extra set of the stream's stored bytes - about fifty per cent more disk, since the base was two - plus one more transfer of every incoming byte, continuously, and a one-off backfill of everything already retained.

An acknowledgement usually means the bytes were accepted into the operating system's file cache, which is volatile memory, not that they were forced onto persistent media. Everything accepted but not yet forced is the loss window a power cut takes.

Copies survive the loss of whatever they do not share: a broker process, a host, a volume. They do not survive what replication reproduces faithfully - a deleted stream, a bad record, expiry, or losing the whole cluster.

Writes to that stream are refused until a third copy is caught up again. Setting the floor equal to the copy count turns the loss of any one copy into a write outage, which is why the floor is normally set one below the count.

No. A newly joined node holds no records, so it serves nothing and relieves nothing until units of ownership are assigned to it and their records have been copied across. Detached-storage designs are the exception.

Writes to that partition fail rather than block: the client's cached owner map still names the dead node, so sends are rejected until a replacement copy is promoted. Records survive only if the writer's retries outlast that gap.

Leadership is a second distribution, separate from where copies sit. Where exactly one node serves each partition (or queue), the work follows leadership rather than storage, so perfectly even copies can still leave one node leading — and serving — most units.

A reassignment plan only declares which node should hold which partition (or queue). Accepting it starts ordinary copy traffic that streams the unit's retained records to the new node, and ownership settles only once that copy has caught up.

A restart takes that node's copies out of service, and while it is down they fall behind. Stopping the next node before they have caught up removes a second current copy and can leave a partition with none.

The arrival rate. Only the surplus between reading rate and arrival rate drains a backlog of unread records, so capacity sized to keep pace in steady state merely holds the backlog at whatever depth the outage left it.

Three events: a member joins, a member leaves cleanly, or a member stops proving it is present — by missing its liveness signal or by holding work past its progress deadline. Designs where readers merely compete reassign nothing.

Unread count states the gap in records; unread age states it in time. Two million records may be seconds old on a fast stream or half a day old on a slow one, so only the age reading shows staleness.

Moving a reader group's read position forward abandons every record between the old position and the new one, so nothing processes them. The gap drops to zero at once because work was skipped, not done.

A slow member still completes records, just fewer than arrive, so its read position keeps advancing. A stuck member completes nothing: its position is parked. Only the slow case is answered with capacity.

Because a large part of what a request costs a node is fixed, whatever the request carries: a round trip, a queue slot, a handler, per-request bookkeeping. A batch spreads that fixed cost over many records, so the same hardware accepts far more records per second.

A saturated node has four responses: queue the work, delay the answer, block the writer, or refuse the request. Each shows up differently at the call site, and the quietest discards records without failing the send.

A byte-rate ceiling caps how many bytes per second the cluster will handle for one named principal - a client identity, user, application or tenant - not per connection. Over it, that principal is either answered more slowly or refused.

A connection cap counts connections held open, not traffic. Holding one consumes memory and bookkeeping on the node whether or not data flows, so a fleet of near-idle clients can exhaust the cap while the byte rate stays tiny.

The node refuses that one write with a size error and keeps serving everything else on the same connection. Retrying the identical record never succeeds: it is a deterministic check, so the record must shrink or a ceiling must rise.

A named space scopes three things: the names streams may take, the grants written against them, and the point a quota attaches to. Nothing physical is scoped — disks, memory, network and node failure stay shared across every space on the cluster.

A stream's name should carry the owning domain, its purpose, the environment and a version segment. Each answers a question a joining team would otherwise have to ask a person, and grants and quotas are written against the leading segments.

Creation on first use turns every typo into a permanent stream: empty, billed, and indistinguishable from a deliberate one. It also means the stream is born with whatever values the platform supplies and no recorded owning team, rather than values anyone chose.

Most platforms have no rename operation — the name is the stream's identity — so a rename means standing up a second stream and moving everyone. Grants, quotas, retention settings, mirroring rules, dashboards and alert conditions all name the old string.

Absence of writes is one line of evidence out of three. A decommission also needs no reader — nothing attached, no stored reading position advancing — and, where records stay readable after delivery, no replay, all observed over a stated observation window and put to the owner record.

A grant names four things: the principal making the request, the operation attempted, the named resource it is attempted on, and whether that combination is permitted or refused. Leave any one loose and the rule decides more than intended.

A client proves itself with a shared secret through a salted challenge-response password exchange, with a client certificate, with a directory-issued or issuer-signed short-lived token, or with delegated platform identity where it holds no secret at all.

Volume-level encryption beneath the broker stops bytes that leave the cluster: a removed drive, a copied volume image, replaced hardware. It stops nobody who connects, because the storage layer decrypts beneath the broker process, which then serves plaintext to any principal holding a grant.

Whole record payloads, their keys and the cluster's coordination traffic. Where copies are kept by nodes forwarding records to each other, that hop carries every record again for each extra copy, continuously, for as long as the cluster runs.

Brokers separate far more than reading and writing: creating a stream, deleting it, changing its settings, listing or describing names, recording a read position, and cluster-wide administration are distinct rights. Teams that grant only two end up granting all of them.

Writes and reads for that partition fail: with no node holding the job of serving it, there is nothing to accept or answer requests. The stream's other partitions keep working, so the outage is a slice, not the whole cluster.

The delivery interval covers the whole path: from a write being accepted to a reader finishing the work on that record. Acknowledgement time covers only the first hop, so the wait in the stream and the reader's own processing are what make up the four minutes.

The p99. Most requests a node serves are cheap, so the mean tracks them and stays flat, while the tail is where a writer actually waits — and one writer in a hundred waiting 700 ms is what a customer reports.

A green cluster dashboard guarantees only that the broker accepted writes, stored them, served reads and recorded that readers moved forward. It says nothing about whether any record was understood, transformed, or turned into useful work downstream.

A usable condition names three things: which signal it reads, the threshold that counts as a breach, and a sustained window the breach must hold across. Size that window longer than one full write-and-drain cycle of the flow it watches.

Because the wire version is settled per connection, not per cluster. At the connect step each side says what it can speak, the connection uses a shape both understand, and the upgraded nodes still answer in that older shape.

The override wins for that one stream; every stream or queue without an override follows the cluster-wide default. The effective value is therefore a per-object answer, and editing the default leaves overridden objects exactly where they were.

Taking one broker node at a time keeps every other member serving, so the cluster stays available through the change. The price is a mixed-version window: until the last node is done, two broker releases are live in the same cluster.

Settings split into live-applying values, which a running broker node re-reads and acts on at once, and restart-only values, which the process reads while starting and holds for its lifetime. A restart-only edit is not ignored — it is pending until the next restart.

Because the agreed internal version fixes how members speak to each other. Held at the old level, a new binary keeps talking the way an old one expects, so any node can still be reverted. Raising it early would leave old members unable to follow their peers.

Writers are pointed at the standby, readers restarted there, and only one site keeps accepting writes. The standby holds only what the copier had carried, and a reader's stored position from the source names a different record there.

A file backup fixes a stream at the instant it was taken, so every record written since is gone, along with every reader's progress. Streams keep moving while files do not, which is why the gap is counted in hours.

A cross-cluster copier is an ordinary client of both clusters: the source stores a record and answers the writer before the copier has even read it. The record therefore exists on the source first and on the target some time later.

The asynchronous copy hop sets the floor. Records acknowledged on the source cluster but not yet carried to the target exist in one place only, so losing the source loses them — the recovery point is at least the copy lag.

An origin stamp: a marker on the record, or an origin-qualified stream name, that tells the copier running the other way this record came from elsewhere. Without one, each copier treats the other's output as new local traffic and the loop never ends.

Hosted broker tiers usually withhold the durability and security internals: how many replica copies a stream keeps, the acknowledgement floor, flush behaviour, and pluggable identity or decision hooks. Stream design, permissions, client versions and the spend stay yours.

Much of a hosted broker's bill is standing rather than usage-driven: purchased throughput capacity reserved ahead of the traffic, the streams, partitions or queues that merely exist, and retained bytes still held, multiplied by their replica copies.

A rental takes over the machines: replacing dead nodes, patching the engine, keeping the endpoint reachable, growing capacity inside the shape you bought. Stream design, retention choice, permissions, client versions, the reading side and the spend stay yours.

A hosted broker's hard caps fall into five classes: how many streams or queues the cluster admits, how many partitions, how many concurrent connections and subscriptions, how large one message may be, and how long history is kept at all.

Habits that are free on owned hardware are billed dimensions once rented: many small streams and a fine-grained partition layout, a generous retention policy multiplied by replica copies, huge populations of tiny records inflating billed request counts, and readers placed across a charged boundary.