Cross-Cluster Continuity

A file backup fixes a stream at the instant it was taken, so every record written since is gone, along with every reader's progress. Streams keep moving while files do not, which is why the gap is counted in hours.

A cross-cluster copier is an ordinary client of both clusters: the source stores a record and answers the writer before the copier has even read it. The record therefore exists on the source first and on the target some time later.

The asynchronous copy hop sets the floor. Records acknowledged on the source cluster but not yet carried to the target exist in one place only, so losing the source loses them — the recovery point is at least the copy lag.

An origin stamp: a marker on the record, or an origin-qualified stream name, that tells the copier running the other way this record came from elsewhere. Without one, each copier treats the other's output as new local traffic and the loop never ends.

Each cluster numbers records itself, so the number names a different record on the other one. A group resumes from a position map the copier maintains, or from a timestamp; both land near the old place, not on it.

Suspect a companion service on the publish path - most often the contract store a producer must reach before it may publish. Broker metrics cannot see it, so the cluster reads green while every write is refused.

A cross-cluster copier carries a record's payload and key unchanged, and preserves order only where one part of the source stream feeds one part of the target in a single stream of writes. The position number is assigned fresh by the target cluster.

The bookkeeping around the records: stored read positions, the entries saying who may connect and act on which stream, per-stream settings that differed from the cluster defaults, and the contract store the payload identifiers resolve against. The records are the easy part.

It exists to make long history affordable and readable through the cluster, not to be restored from. The parked objects are addressed through the cluster's own bookkeeping, so on their own they are opaque files with no reader positions, permissions or settings attached.

Only a position map written by the cross-cluster copier while it ran: sampled pairs of a source position and the nearest position on the target. Without one, a number recorded against the source names no particular record on the target.

Readers dominate the recovery time, not the cluster. A standby fed by an ongoing copy is already running; the clock is spent repointing readers and writers, re-establishing where each reader resumes, and working off everything that piled up while nothing was consuming.

There was never one order to recover: each site sequenced only its own writes, and nothing anywhere recorded how the two interleaved. Any merge has to invent a rule — write timestamps, site priority — and different rules produce different, equally unfounded answers.

Establish that the cluster is genuinely unusable rather than merely unreachable, how far behind the standby is now, that producers can be kept off the source, and that every reader group has a defensible start position on the standby.

The roles have reversed: the standby now holds the only copy of a day's records, while the original holds a stale history plus a leftover tail. Returning means running the copy the other way first, then translating reader positions again.

Choose per stream, by what a repeat and a miss each cost downstream. Default to landing early and reprocessing, since a duplicate is usually recoverable and an unread record is not, then make the overlap wider than the restart's uncertainty.

A contract store keeps state of its own, and where it lives decides the restore: on the cluster it serves, in another team's database, or in a checked-in definition. Only the last comes back for free.

The two clusters number their streams independently, so that subtraction is arithmetic on unrelated counters and is meaningless even when the hop is perfectly caught up. Measure inside the source's own space, or in seconds of record age.

Those records were acknowledged to their writers, so upstream believes the facts happened while every system derived from the stream is permanently short of them — and the gap is silent, because readers resuming on the target cluster see an unbroken stream with no hole in it.

One stream written at two sites has no defined order across them and needs loop prevention on every hop just to exist. Giving each site its own stream with one owning writer, read everywhere, removes both problems and moves the combining work onto readers.

Inventory everything a publish and a read touch besides the brokers, then give each companion a posture: one shared instance, a copy per site, or rebuild on demand. Anything excluded should be excluded on purpose, in writing.

An unrehearsed target is a stated intention, not a capability. Only a real switch measures the stages nobody budgeted — access on the standby, re-establishing reader resume points, the decision latency, the catch-up — so rehearse it, record the measured numbers beside the stated ones, and revise whichever is wrong.