Before rewinding a reader group's read position by six hours to reprocess, what side effects must an operator account for first?

## The move is cheap; the consequences are not Moving a reader group's recorded read position backward is a single administrative write. What it triggers is a full re-run of everything the group does for the span between the new position and where it was — at whatever rate the readers can manage, which is usually far faster than the rate the records originally arrived at. The question an operator has to answer before the move is therefore not "can we rewind" but **"what fires again, and which of it escapes?"** ## The inventory: three classes of effect | class | examples | what a rewind costs | |---|---|---| | **Convergent, inside the service** | a row written by key, a value set to the latest seen, a cache entry rewritten | usually nothing but CPU: the second run lands on the same state | | **Accumulating, inside the service** | counters that add, rows appended to a history, totals summed | silent corruption: the span is counted twice unless the accumulation is keyed against a repeat | | **Escaping the service** | email, SMS and push, payment capture or refund, calls into a partner, physical fulfilment, records published onward | not recoverable by the readers at all: the effect has already left | The third row is the one that turns a rewind into an incident, and it is the class that a well-written handler cannot rescue. Making the handler absorb a repeat is a real and separate discipline, and it operates on state the service owns. It has no reach over a message that has already been delivered to a customer or an instruction already accepted by a payment provider. ## The effect that is most often forgotten A reader that **publishes records onward** — enriching, splitting, or routing them to another stream — makes every downstream reader group a participant in your rewind without being asked. They receive the re-published span as ordinary new work, and their own escaping effects fire too. One team's six-hour rewind becomes four teams' six-hour rewind, discovered by the others from their own incident channels. This is why the inventory has to follow the records, not the service: enumerate what is downstream of the new position across the boundary, and tell the owners before the readers restart, not after. ## Containment, in the order it is decided 1. **Bound the span.** Rewind to the narrowest range that fixes the problem, not to a comfortable round number. Every extra hour is extra duplicate effect for no benefit. 2. **Disable the escaping leg.** Turn off the outbound path — the notification sender, the payment call, the partner client — for the duration, or point it at a sink. This is an operational change made before the readers restart, and it is the single most effective control available. 3. **Throttle the re-run.** The rewound span arrives at the downstream dependencies as a burst. A partner's rate limit that is comfortable at the natural arrival rate will be exceeded by a replay running an order of magnitude faster, and rejections during a recovery are their own incident. 4. **Warn the neighbours.** Anyone whose reader group sits downstream of a republished stream, and anyone who owns an accumulating total, needs to know the span and the time window. 5. **Agree who owns the residue.** Duplicate customer-visible effects that escape containment need an owner and a story before the move, not an apology afterwards. ## Making the decision honestly A rewind is worth its cost when the missing or wrong output is worth more than the duplicate effects it will produce — a derived total that is wrong for six hours, a batch of records dropped by a broken handler. It is a poor trade when the only convergent part is cheap and the escaping part is large: re-sending a day of notifications to fix a reporting number is a worse outcome than the number. Where the effects are heavily escaping, the alternative is to reprocess **without the live path**: give the work a separate reader group with the escaping legs disabled, and reconcile the difference by hand. That keeps the live group's position untouched and turns a broadcast into a measurement. ## Where the move is not available All of this assumes a design that keeps records after they are read and keeps a position that can be written. Where records are deleted once acknowledged, there is nothing to rewind into: the only source of a second pass is a copy written elsewhere when the records were first produced, which is a decision made long before the incident. ## What an interviewer is listening for Not "make the consumer idempotent" — that answers a different question and is assumed here. The signal is whether the candidate separates effects that converge from effects that escape, remembers the republished stream, and treats disabling the outbound leg and bounding the span as steps taken **before** the position is written rather than as mitigations discovered afterwards.