A stream keeps three copies within a single cluster. Which failures does that survive, and which does it not?
answer
- redundancy against what, exactly?
- what do the copies share?
- replication reproduces deletes faithfully
- retention and cluster loss hit every copy
basics
~20 sCopies survive the loss of whatever they do not share: a broker process, a host, a volume. They do not survive what replication reproduces faithfully - a deleted stream, a bad record, expiry, or losing the whole cluster.
solid answer
~40 sKeeping three copies of a stream inside one cluster buys protection against the *independent* loss of the things those copies sit on. If each copy is held by a different node on a different host and a different volume, the cluster can lose a broker process, a host or a disk and still serve the stream from a survivor. It buys nothing against anything the copies reproduce faithfully: a stream someone deleted, a malformed record a writer sent, records already removed by the retention rule, or a failure that takes every host holding a copy at once. And the number alone is only a promise about how many *separate* things must fail - a promise that holds only if the copies really are separate.
go deeper
Recall the trade in one line: more copies survive more hardware and process failures, and survive nothing that the cluster is correctly told to do. Naming one thing extra copies do not help with is what gets you the mark.
Explain the condition attached to every case: a copy protects only against failures of things it does not share with the other copies, which is why the number and the placement have to be quoted together.
Show the check you run in production: what the copies are separated across, whether this stream carries the count you assume, and what sits underneath the nodes. Say plainly that retention and deletes reach every copy.
Frame it as the depth of a guarantee whose axis is placement, and price it: every extra copy multiplies stored bytes and cross-cluster traffic, so the estate needs a small number of deliberate levels rather than one number applied everywhere.
## What a copy is, and what the count promises A **copy** is one full stored replica of a stream's data held by one node of the cluster. The **copy count** is how many of those the cluster keeps for that stream. Platforms differ in how the copies relate to one another: on many, one copy leads and the others pull from it; on others a write is committed by a majority of the copies with no single follower relationship; and on designs where durability comes from a shared durable store underneath, a broker-held copy may not be where the authoritative bytes live at all. What is common to every shape is the promise the number makes, and it is narrower than it sounds: > Three copies means **three separate things must fail** before the stream is unreadable - provided the three really are separate. Everything below follows from that one sentence. ## The failures extra copies survive - **A broker process dying or being restarted.** Another node already holds the data and can serve it. - **The loss of a host** - power supply, kernel panic, a motherboard - *only if* the copies are not all on that host. - **A volume failing or corrupting** - *only if* the copies do not share that volume, or a storage array behind it. - **Planned maintenance.** A node taken out for a reboot or an upgrade is the same event as a crash, just scheduled; surviving copies are what make it a daytime task rather than an outage. - **A wider domain - a rack, a power feed, an availability zone** - *only when* a placement rule deliberately put copies in different ones. Notice the pattern: every entry carries the same condition. A copy protects against exactly the failures of the things it does **not** share with the other copies. ## The failures extra copies do not survive - **Anything the cluster was correctly told to do.** A deleted stream is deleted in every copy; a truncation applies everywhere. Replication is obedient, not sceptical. - **Bad data.** A malformed, duplicated or simply wrong record is copied faithfully to every replica within milliseconds. - **Age.** When the retention rule removes a record it removes it from every copy, because retention is a property of the stream rather than of one node's disk. - **Correlated failure.** If all three copies are in one rack, one power event is one failure, not three. - **Loss of the whole cluster.** Everything here is inside one cluster; surviving the loss of the cluster itself is a different mechanism entirely. - **Records that only ever reached one node.** The count says how many copies *exist*; it does not by itself say how many had to accept a write before the writer was told it succeeded. That is a separate rule, and a three-copy stream can still be answering writes only one node holds. | Failure | Survived by three copies? | Condition | |---|---|---| | One broker process dies | Yes | another copy is current enough to take over | | One host is lost | Yes | the copies are not all on that host | | One volume fails | Yes | the copies do not share a volume or array | | A rack or zone loses power | Only sometimes | a placement rule separated the copies | | An operator deletes the stream | No | the delete is applied to every copy | | Retention removes old records | No | retention applies to the stream itself | | The whole cluster is lost | No | needs something outside this cluster | ## Why "we keep three copies" is not yet an answer Three facts turn the number into a real durability statement, and an interviewer is usually listening for at least the first: 1. **What the copies are separated across.** Three copies on one host is one failure wearing three hats. The number is a *depth*; placement is the *axis*. 2. **Whether this stream really has that number.** The count is chosen per stream, usually from a cluster-wide default that applied on the day it was created - so "our standard is three" and "this stream keeps three" are different claims. 3. **What it costs on the wire.** Each copy is another full stored replica, so one incoming byte becomes three stored bytes and roughly two transferred - this is **write amplification**, and it is the reason the count is not simply set as high as it will go. ## How to answer it out loud State the rule, not the number: extra copies survive the failure of anything the copies do not share, and survive nothing the cluster faithfully reproduces or that takes all of them at once. Then say what you would check - where the copies sit, and whether this particular stream actually has the count you think it has. A candidate who says "three copies, so we cannot lose data" has answered a different, easier, wrong question.
- Does adding a fourth copy of a stream inside one cluster make an accidental delete recoverable?No. A delete is an instruction about the stream, and every copy applies it; so does the retention rule. Copies protect against hardware and process failure, not against an instruction the cluster carried out correctly. Recovering from a wrong instruction needs something outside this cluster's own replication.
- Two teams both say they keep three copies. What would you ask to find out whether the claims mean the same thing?Three things: which failure domains the copies are separated across, whether this stream's own count really is three or only the cluster default is, and what sits under the nodes - two volumes carved from one array, or two broker nodes scheduled onto one physical machine, make the number a fiction.
- What does each extra copy cost, leaving money aside?Traffic and write amplification. One incoming byte becomes N stored bytes and roughly N-1 bytes crossing the cluster to the other copies, and a newly placed or lagging copy pulls catch-up traffic on top of the steady stream. That bandwidth competes with the writes and reads the cluster is there to serve.
saying these in an interview costs you the question
- Says three copies mean records cannot be lost.
- Treats copies inside one cluster as a backup.
- Believes replication protects against a mistaken delete.
- Assumes copies are automatically on different hosts.
- Quotes a copy count without saying what it separates.