Operating Signals & Alerting

The delivery interval covers the whole path: from a write being accepted to a reader finishing the work on that record. Acknowledgement time covers only the first hop, so the wait in the stream and the reader's own processing are what make up the four minutes.

The p99. Most requests a node serves are cheap, so the mean tracks them and stays flat, while the tail is where a writer actually waits — and one writer in a hundred waiting 700 ms is what a customer reports.

A green cluster dashboard guarantees only that the broker accepted writes, stored them, served reads and recorded that readers moved forward. It says nothing about whether any record was understood, transformed, or turned into useful work downstream.

A usable condition names three things: which signal it reads, the threshold that counts as a breach, and a sustained window the breach must hold across. Size that window longer than one full write-and-drain cycle of the flow it watches.

Those partitions are still being served but now survive fewer failures: copies that should be current are not keeping up. A brief spike during a change is normal; an hour with no change in progress means something is not catching up.

Wait time is how long a request sat in the node's request queue before a handler took it; service time is the handling itself. Rising wait means too little handler capacity for the arrival rate; rising service means the work got slower.

Near-zero reader record lag says records are being read and progress is being recorded — not that the work succeeded. A reader can move past every record while the processing throws, writes nowhere, or is rejected downstream.

Because a percentage says nothing about how long you have. The same 80% is months away on a quiet stream and an hour away on a busy one; a projection from the measured fill rate turns the reading into a deadline you can act on.

A write-side timestamp puts the writer's own accumulation, batching and retries inside the measured interval and trusts a clock you do not control. An arrival timestamp is set by the cluster on acceptance, so it is comparable across records but hides everything that happened before the record was accepted.

Because writes, reads and metadata calls do unrelated work with unrelated costs. One percentile over all of them moves when the traffic mix changes, not only when something slows, and it never says which side of the node is hurting.

Raising the threshold trades away the detection the alert existed for at every other hour. Split the signals a planned change is expected to move from those it never should, size the sustained window past the normal catch-up time, and bound any suppression to the announced change window.

Alert on the shape of the curve rather than its height: fire when the unread backlog fails to fall across a sustained window, when it is still above a level at a deadline the owner states, or when the reader's stored read position stops advancing.

Because the cluster has lost the ability to change, not to serve. Existing assignments keep working, so request dashboards stay green, while the next node failure cannot be repaired — leaving the cluster one ordinary fault away from unserved partitions.

Because when a cluster signal is bad, every downstream number is a consequence rather than a cause. Reading the reader-side numbers first sends investigation to the wrong teams for what is really one unserved partition, one lagging copy set, or a frozen coordination plane.

The difference between two hosts' clocks, plus the real interval. A negative result proves the offset between the writing host and the reading host is larger than the journey itself, so every sample from that pair is wrong by that offset — the small ones ruinously.

The same client path, the same cluster, the same acceptance rules and coverage of every part of the stream — written and observed by one host so the timing needs no cross-host subtraction. Anything the probe does not share is a segment it silently excludes.

Because occupancy fills before waiting appears. Duration stays flat while a pool has spare handlers and then rises steeply once it does not, so saturation and request-queue depth usually turn first and buy lead time that latency does not.

Anything that happens after hand-off is invisible: a reader recording progress without doing the work, a transform writing its output nowhere, a destination rejecting every write, a filter dropping everything. The cluster observes transport, not consequences.

The honest signal is a count of completed business outcomes, published by the team that owns the pipeline, because only that team can define completion. Requiring it estate-wide costs per-pipeline instrumentation, extra stored series, and a definition argument per pipeline.

Conditions on the cluster's own health are properties of the cluster, so one number serves the estate and the platform team owns it. Conditions on flow are properties of a stream's commitment, so the owner sets the number; an unset stream gets a non-paging default.