Why can't copy lag between two clusters be measured by subtracting the target stream's newest position number from the source's?
answer
- two counters, one subtraction
- stay inside the source's numbering
- records and seconds answer differently
- the idle stream lies in seconds
basics
~20 sThe two clusters number their streams independently, so that subtraction is arithmetic on unrelated counters and is meaningless even when the hop is perfectly caught up. Measure inside the source's own space, or in seconds of record age.
solid answer
~50 sThe target assigns its own position numbers as it appends, so the source's newest number and the target's newest number are coordinates in two different spaces. Subtracting them can read zero on a stalled hop or a large constant on a healthy one. Two honest measurements exist. **In records:** compare the source position the copier has read up to against the newest position on the source - both numbers live in the source's space, so the difference is real. **In seconds:** compare the source timestamp of the newest record the copier has carried against now, which requires that the copier preserves source timestamps rather than stamping arrival. The two units answer different questions and disagree under load and at idle, so operators usually watch both, per part of the stream, and care about the worst part rather than the average.
go deeper
Recall that the two clusters number their streams separately, so comparing their newest numbers is not a measurement of anything.
Be able to state the correct computation: the copier's read progress against the newest position, both in the source cluster's own numbering, plus a seconds-behind figure from record timestamps.
Show the operating instincts - per part and worst-case rather than average, both units because they disagree at idle and under bursts, and a deliberate low-rate record to prove the hop is alive.
The point to press is that a number nobody can interpret is worse than none: an estate that reports copy lag wrongly will believe it is protected and discover otherwise only when it needs the far side.
## Why the subtraction is meaningless A record's position number is issued by the cluster that appended it. The stream on the **target cluster** is an independent append-only sequence with its own numbering, so 'newest number on the source minus newest number on the target' subtracts two counters that were never in the same system. The result has no unit and no interpretation. Worse, it fails in both directions: - a hop that started after the source stream did shows a large difference **forever**, even when perfectly caught up; - a target stream that already held records, or that has absorbed duplicates from copier retries, can show **zero or a negative** difference while the hop is dead. A number that is large when things are fine and small when they are broken is worse than no number. ## Two measurements that do work | measurement | how it is computed | what it answers | what it needs | how it misleads | |---|---|---|---|---| | records behind | newest position on the source minus the source position the copier has read up to - both in the source's own space | how much stream is not yet copied | the copier's read progress, per part | says nothing about how old the gap is; a slow trickle of large records looks small | | seconds behind | now, minus the source timestamp of the newest record carried across | how stale the far side is | the copier preserving source timestamps rather than stamping arrival | on an idle stream it grows while nothing is actually missing | The key move in the first row is that **both** numbers are the source's own. You are not comparing two clusters; you are comparing the copier's progress against the stream it is reading. That is a legitimate subtraction. ## Why both units, and why they disagree 1. **Bursts.** A write burst pushes 'records behind' up sharply while 'seconds behind' barely moves, because the copier is still carrying recent records - just a lot of them. 2. **Idle streams.** A stream nobody writes to shows zero records behind and a 'seconds behind' figure that climbs forever, because the newest record carried is genuinely old. Nothing is wrong. Teams that watch only the seconds figure chase this ghost regularly. 3. **Record size.** Records behind is a count, not bytes. A hop limited by bandwidth drains a thousand large records far more slowly than a thousand small ones, and only the seconds figure shows it. ## Measure per part, and report the worst On platforms that split a stream into parts, the copier tracks progress per part and the hop can be healthy on nineteen parts and stalled on one. An average over parts hides exactly the failure you care about, because the stalled part is the one whose records are missing on the far side. Report the **maximum across parts**, and keep the per-part detail so the stalled one can be named. ## The measurement everybody forgets: is the hop alive? Both figures above are computed from what the copier reports about itself. That leaves a blind spot: - a **stopped copier** that no longer reports leaves a stale figure on the dashboard, which can look like a steady healthy number; - an **idle stream** makes a dead hop and a healthy hop look identical, because neither is carrying anything. The usual answer is to give the hop something to carry on purpose: a low-rate record written into a stream that the hop copies, and observed on the far side. If it stops arriving, the hop is dead, regardless of what the lag figures say. That check is about the copier's liveness, and it is separate from the two size figures. ## Where platforms differ - Where readers and copiers track a numeric position per part, records-behind is natural and cheap. - Where a platform exposes only time-based progress, the seconds figure is all you have, and it inherits the idle-stream artefact described above. - On queue-shaped platforms with no stored numeric position, a hop's backlog is usually visible as the depth of whatever the copier is consuming from, which is a count of messages rather than a distance between two coordinates. - What to do with either number - what loss you are willing to state, and what wakes somebody up - is a separate decision from measuring it correctly, and it can only be made once the number means something.
- Copy lag between the two clusters reads zero in records and forty minutes in seconds. What is happening?Almost certainly an idle stream: the copier has carried everything that exists, and the newest record it carried is forty minutes old because nothing newer has been written. Nothing is missing. The seconds figure measures staleness of content, so it climbs whenever writers go quiet.
- Why report the worst part rather than the average across parts of the stream?Because a hop stalled on one part of the stream still leaves those records absent on the far side, and an average over twenty parts dilutes that to near-invisibility. The maximum names the actual exposure, and the per-part breakdown tells you which part to investigate.
- How do you tell a healthy hop on a quiet stream from a hop that has stopped entirely?Give it something to carry. A low-rate record written deliberately into a copied stream and observed on the target proves the whole path end to end. Without it, both situations produce the same flat numbers, and a dashboard fed by a stopped copier can look reassuringly steady.
saying these in an interview costs you the question
- Computes copy lag by subtracting the newest position numbers of the two clusters
- Assumes a low records-behind figure proves the hop is running at all
- Reports an average across parts of the stream rather than the worst part
- Confuses copy lag between the two clusters with how far a reader trails the newest record
- Reads a climbing seconds-behind figure on an idle stream as a stalled hop
- Assumes the seconds figure is available even when the copier stamps arrival time