skip to content

Dual publication writes every record to both the old stream and its replacement during a cut-over: what does it protect, and what does it not?

level: seniorimportance: should knowfreq 48%

answer

  1. copies records, not bookkeeping
  2. either stream is a complete source
  3. gap or overlap, pick one
  4. positions belong to one stream
  5. protection ends when publication stops

basics

~20 s

Dual publication protects whoever has not been moved yet: both streams carry the full traffic, so consumers can be relocated gradually and put back. It does not carry stored positions across, does not remove duplicates, and protects nothing after the delete.

solid answer

~50 s

What it buys is time and reversibility. For the length of the cut-over window either stream is a complete source, so readers and writers move independently, a consumer discovered late is still working, and any move can be undone by pointing the client back. What it does not buy is continuity of progress. A reading group's stored position is meaningful only against the stream it was recorded on, so a group moved to the replacement starts where you deliberately place it — at the replacement's beginning, reprocessing records it already handled, or at its end, skipping whatever arrived in between. It never resumes "where it was". It also does not stop a consumer attached to both streams from acting on the same record twice, and on a platform where a delivered message is gone there is no stored position to argue about at all.

go deeper

for a junior

Remember the split: both streams get every record, so nobody goes hungry during the move, but the new stream has no idea how far anyone had already got on the old one.

for a middle

Explain why a stored position cannot travel — it is a coordinate in one stream — and state the two placement options and the failure each one produces.

for a senior

Demonstrate the judgment: choose gap or overlap per consumer from what its work costs when repeated, plan the backfill when you choose the gap, and name the double-processing hazard a two-stream window creates.

for a principal

Frame it as a cost decision: the window's duplicate storage is the premium paid for reversibility, and the question for the estate is how long that premium is worth paying rather than whether to pay it at all.

## What dual publication buys During a retirement there is a period when the old stream and the **replacement stream** both receive every record. That period is the **cut-over window**, and the practice of writing to both is **dual publication**. Its purpose is to make one hard, atomic-looking change — "everything moves to the new stream" — into a series of small, independent, reversible ones. Concretely it buys four things: - **Independence.** Readers and writers no longer have to move in the same change, or even the same week. - **Granularity.** Reading groups move one at a time, each verified before the next is touched. - **Reversibility.** A move that goes badly is undone by pointing the client back at the stream it came from, which still has everything. - **Tolerance of ignorance.** A consumer nobody remembered keeps working, because the stream it reads is still being fed. A missed consumer becomes a scheduling problem instead of an outage. That last one is the real reason the practice exists. Estates do not have perfect knowledge of who reads what, and dual publication converts that imperfection from a hazard into a delay. ## What it does not carry: progress Dual publication copies **records**. It does not copy the bookkeeping that says how far a consumer had got. Where a reader owns a stored position, that position is a coordinate in one stream. The same record occupies a different coordinate in the replacement, and nothing reconciles the two. So a reading group moved during a cut-over does not resume; it is **placed**. There are exactly two honest choices, and they have opposite failure modes: | Where you place it | What you get | What it costs | |---|---|---| | At the replacement's earliest available record | No gap: everything the replacement holds is processed | Overlap: records already handled on the old stream are processed again | | At the replacement's newest record | No repeats | A gap: whatever arrived between the start of the window and the move is never read from either stream | There is no third option that gives neither, because there is no correspondence between the two streams' positions to exploit. Which one you pick is a property of the consumer, not of the retirement: work that is safe to repeat takes the overlap, work that must not be repeated takes the gap and someone backfills it deliberately from another source. On platforms where a **delivered message is gone**, the question dissolves in a different direction: there is no stored position to move, and the equivalent decision is whether the old stream is drained before its writers leave, or whether whatever is left in it at the end is simply accepted as lost. ## The duplicate the window creates For the length of the window, each business event exists twice in the estate — once on each stream. That is harmless **only** while every consumer reads exactly one of them. A consumer attached to both, or a consumer moved without being detached from the old stream, performs its work twice. Dual publication offers no protection whatsoever here: it is a publishing technique, not a deduplication mechanism, and whether double processing is survivable is a property of the consumer, decided long before this retirement started. The same duplication is a cost: the records are stored twice and, on a rented cluster, usually charged twice. That cost is the budget for the whole safety property, and it is small compared with a deleted stream, which is why an extended window is nearly always the right answer when something unexpected turns up. ## What it stops protecting, and when The protection has a clean end date. The moment dual publication stops, the old stream receives nothing. From then on: - A consumer still attached to the old stream gets silence, not traffic. That is the warning the quiet period is for. - On a log-shaped platform, what the old stream already holds stays readable until the delete, so a late discovery can still read history there. - On a queue-shaped platform, there was never much left behind to read. - After the delete, none of this applies — there is no stream to point anything at, and the question becomes where else the data exists. ## Where platforms differ The practice is portable; the detail is not. Where a stream is split into parts and readers own a rewindable numeric position, placement is an explicit operational choice and the gap-or-overlap table above is exactly the decision. Where consumers compete for messages that vanish on acknowledgement, the move is about draining rather than placing. Some platforms let a consumer be attached to two streams effortlessly, which makes the double-processing hazard easier to hit. The single claim that holds everywhere is the one worth remembering: **both streams carry the records, and neither carries the progress**.

  • A consumer must not process any record twice and must not miss one. How do you move it during a cut-over?
    You cannot get both from placement alone. Place it at the replacement's newest record, which gives a clean no-repeat start, and close the gap deliberately: identify the interval between the window opening and the move, and backfill that work from whatever other record of it exists. Treat the backfill as part of the retirement plan, not as an incident afterwards.
  • What makes an extended cut-over window cheap, and what eventually makes it expensive?
    It is cheap because the only ongoing cost is storing and, on a rented cluster, paying for each record twice. It becomes expensive when it stops being temporary: writers now have two paths to maintain, every new consumer must be told which stream is canonical, and the estate quietly grows a permanent duplicate. Extend for a known reason and a known date.
  • Does dual publication guarantee the two streams hold the records in the same order?
    No, and nothing in a retirement should depend on it. Each publication is its own write, and the ordering a platform guarantees applies within a stream, not across two. Consumers whose correctness depends on relative order must be moved at a boundary where that order does not straddle the cut.

saying these in an interview costs you the question

  • Assumes a moved reading group resumes where it left off
  • Thinks dual publication removes duplicates for consumers
  • Expects the two streams to share position numbering
  • Leaves a consumer attached to both streams after moving it
  • Keeps dual publication running indefinitely with no end date