skip to content

Why does a reader group's stored read position from the source cluster not identify the same record on the standby, and how does it resume there?

level: middleimportance: must knowfreq 62%

answer

  1. each cluster numbers records itself
  2. the number exists but names another record
  3. a map the copier writes as it runs
  4. or restart from a moment in time
  5. nearest, not identical — replay or gap

basics

~20 s

Each cluster numbers records itself, so the number names a different record on the other one. A group resumes from a position map the copier maintains, or from a timestamp; both land near the old place, not on it.

solid answer

~50 s

A stored read position is a number in one cluster's own numbering of a stream. The standby received the records through a copy hop and numbered them itself, so the same number on the target names a different record — or no record at all. Two mechanisms bridge the gap. A **position map** is a correspondence the cross-cluster copier writes as it runs, recording roughly where a source position lands on the target; restarting means looking the group's positions up in it. A **timestamp** restart instead asks the target for the first record at or after a chosen moment, which needs no map but relies on record timestamps being meaningful. Neither is exact: both land on the nearest carried record, so the group either sees a little it has already handled or misses a little.

code

yaml · 17 lines
yaml
reader_group: orders-fulfilment
source_cluster: site-a
target_cluster: site-b
mapped_at: 2026-09-20T11:04:09Z
entries:
  - stream: orders
    partition: 0
    source_position: 918447201
    nearest_target_position: 41882336
  - stream: orders
    partition: 1
    source_position: 874120044
    nearest_target_position: 39004918
  - stream: orders
    partition: 2
    source_position: 901337850
    nearest_target_position: 40551207

go deeper

for a junior

Recall that a position number belongs to one cluster's own numbering of a stream, so it does not point at the same record anywhere else, even though the records themselves were copied across.

for a middle

Explain both resume mechanisms and what each needs: a position map written by the copier as it runs, or a restart from a moment in time. Say why neither lands exactly on the old place.

for a senior

Show the operational consequence: the restart is always slightly early or slightly late, that is a per-stream decision, and the reprocessing it causes is usually the longest part of the switch.

for a principal

Treat the map, its location and the meaning of record timestamps as things that must be established and rehearsed before an incident, since all three are unavailable for verification at the moment they matter.

## Why the number does not travel A **stored read position** is where a **reader group** resumes after a restart. On platforms that split a stream into parts and let the reader own a position, that position is an integer in the cluster's own numbering for that part of that stream. The standby did not receive that numbering. It received **records**, written into its own streams by a **cross-cluster copier** acting as an ordinary producer, and it numbered them as it appended them. Several ordinary things make the two numberings diverge: - the target's stream did not start empty or did not start at the same moment; - the copier was started, stopped or restarted at some point and re-read a little; - some records were retired from the source by its retention rules before the copier reached them or after it did; - the copier writes into the target in its own batches, so a one-to-one correspondence exists only where it happens to. So position 918,447,201 on the source and position 918,447,201 on the target are simply two unrelated records. Worse, the number usually *exists* on the target, so nothing errors — a group restarted naively just starts reading from an arbitrary place, which is how a switch silently skips or re-reads days of a stream. ## The two ways to resume ### A position map The copier can write, as it runs, a record of the correspondence it is creating: for each stream and part, roughly which target position a given source position landed at. At restart you look each group's stored positions up in that map and start the group there. - **Strength:** it is derived from the actual copying, so it follows the stream's real structure. - **Limit:** it is sampled, not per-record — the entry you find is the nearest one at or before your position, so you start a little earlier than you were. - **Prerequisite:** the map has to have been written all along, and it has to have survived. A map living only on the cluster you just lost is no use. ### A timestamp Most brokers can answer "give me the first record in this stream at or after this moment". If you know roughly when the group had got to — because it was reporting a lag figure, or because the last record it handled carried a time — you can restart every group on the standby from that moment. - **Strength:** it needs no extra machinery and works for groups nobody prepared for. - **Limit:** it is only as good as the timestamps. If the time on a record is the moment the **copier** wrote it rather than the moment the producer did, then every record on the standby appears to have been written during the copy, and a timestamp restart lands somewhere meaningless. - **Prerequisite:** know which of those two times your records carry before the incident, not during it. | | position map | timestamp | |---|---|---| | set up in advance | yes, by the copier | no | | accuracy | nearest mapped entry before you | nearest record at or after a moment | | fails when | the map was not kept or is lost | record times are copy times, not write times | | usable for an unprepared group | no | yes | ## What this means operationally Because neither mechanism is exact, the restart is always slightly wrong in one of two directions, and which direction is a decision rather than an accident. Landing early means the group re-reads records it has already handled; landing late means it never sees a few. Restarting a group is therefore not a mechanical step you can hand to a script without first deciding, per stream, which of those two is acceptable. A second consequence: if a group's positions can only be recovered approximately, the *time* the switch takes includes the group working through whatever it re-reads. That is frequently the longest part of a switch, and it scales with how far back you chose to land. ## Where platforms differ - Where the reader owns a numeric, rewindable position, everything above applies directly. - On **destructive-read** designs, where a message is removed once acknowledged, there is no stored position to translate at all. What crosses is the set of messages the copier carried; anything already acknowledged on the source was never a candidate, and the restart question becomes "is the standby's backlog the right one" rather than "where do we resume". - Some platforms keep a reader's progress as a per-subscriber acknowledgement state rather than a single number per part; the translation problem is the same shape, but it is per subscriber and there may be several to re-establish for one stream. - Where the copier is supplied by the platform, a map may be produced for you. Verify it exists and is readable from the standby before you rely on it, because that is precisely the moment the source is unavailable.

  • The group's stored position number happens to exist on the standby. Why is that worse than it not existing?
    Because nothing fails. A missing position produces an error or an out-of-range fallback someone notices; a valid one produces a group that quietly begins reading from an unrelated place. Depending on which way it lands, the switch silently reprocesses or silently skips a large span of the stream, and the first evidence is usually downstream.
  • What makes a timestamp restart untrustworthy on a standby?
    If the time carried by a record on the target is the moment the copier appended it rather than the moment the original producer wrote it, every record looks as though it was written during the copy. A restart from a business moment then lands arbitrarily. Establish which time your records carry before you need it.
  • Does a position map have to be kept somewhere other than the source cluster?
    Yes, in practice. A map that exists only on the cluster you have just lost is unavailable exactly when it is needed. Copiers that maintain one usually write it into the receiving cluster for that reason; if yours writes it somewhere else, confirm the standby can read it without the source.

saying these in an interview costs you the question

  • Assumes the two clusters number the same record identically
  • Thinks the copier carries reader progress along with the records
  • Believes a position map gives an exact per-record correspondence
  • Trusts record timestamps without knowing which write they mark
  • Expects a restart at a stale position number to fail loudly
  • Treats reader restart as a scripted step needing no per-stream decision