skip to content

Should an estate move to detached storage so that adding or removing a cluster node stops costing copy traffic at all?

level: principalimportance: should knowfreq 38%

answer

  1. the cost it removes must be one you pay
  2. capacity follows ownership, not data
  3. one store every node depends on
  4. the hot tail is usually still local
  5. elastic clusters yes, fixed clusters no

basics

~20 s

Sometimes. Detaching storage turns every membership change from a data movement of hours into an ownership handover of seconds, which is worth a great deal to an elastic estate — but it makes one shared store a dependency of every node, and it rarely removes the recent records that still live locally.

solid answer

~50 s

The question is whether your estate's pain is membership elasticity. Where records live on shared or remote storage, a joining node can serve as soon as ownership moves to it, and a leaving node is handed over rather than copied off, so scaling becomes a control-plane decision of seconds instead of a scheduled overnight change. That is transformative for clusters that must follow a daily traffic curve, and close to irrelevant for a fixed cluster that changes size twice a year. Against it: every node now depends on one shared store whose availability and latency bound the whole cluster, the write path usually gains latency, the charge model moves from disks you own to requests you make, and most designs still keep the most recent records locally — so the hot tail still moves when a node does. It is a platform decision, not a tuning one.

go deeper

for a junior

Know the distinction behind the debate: when a node's records sit on storage any node can read, moving ownership does not mean moving the data, so growing a cluster stops being a copying job.

for a middle

Be able to say what still happens — the handover, the records not yet pushed to the shared store, the cold cache on the new owner — rather than treating the change as cost-free.

for a senior

Argue it from operations: how often your clusters actually change size, what a change costs today, and what a shared store being slow would do to every cluster at the same moment.

for a principal

Deliver a position, not a preference: which clusters are declared elastic, what an outage of the shared dependency is allowed to cost, what the payback assumptions are, and what evidence would reverse it.

## What the choice actually is Two shapes exist in this product class, and almost everything about membership change follows from which one you are on. - **Records on local disk.** Each node keeps its own complete stored duplicate — its **copy** — of each unit of ownership it holds. A unit of ownership is a partition (or queue) that exactly one node serves at a time. Adding a node means filling copies; removing one means emptying them. Both are data movements measured in bytes over available bandwidth. - **Detached storage.** A unit's records live on shared or remote storage that any node can read. Ownership of a unit can be handed from one node to another without moving the records at all. The principal's question is not which is technically nicer. It is whether the cost that detaching storage removes is a cost this estate is actually paying. ## What it genuinely changes | Membership event | Records on local disk | Detached storage | |---|---|---| | A node joins | fill copies; hours; more load first | ownership moves; seconds | | A node is evacuated | move every unit's records off first | hand ownership over; settle what is unpushed | | Expanding during an incident | makes it worse before better | a usable mitigation | | Replacing a failed machine | rebuild its copies from the others | give its units to surviving nodes | | Following a daily traffic curve | not realistic | routine | The second column is the subject of this whole leaf; the third is the subject disappearing. That is a real reduction in operational surface, and it is the honest case for the move. ## What it does not remove 1. **The evacuation itself.** A leaving node still hands its units over and still has a completion condition. It is short, not absent. 2. **The recent records.** Most designs of this kind keep newly written records on the node until they are pushed to the shared store — for write latency, for ordering, or because the shared store is not good at small appends. So a membership change still moves the hot tail, and "no copying" means "much less copying". 3. **Cold caches.** A node that takes ownership instantly still has to read from the shared store on the first requests, so the first minutes after a handover are slower even though the handover was fast. 4. **Redundancy decisions.** How many durable copies exist and where becomes a property of the shared store rather than of your node layout, which is a different conversation — not a removed one. ## What it costs - **A dependency every node shares.** The shared store's availability and latency now bound the cluster's. A local-disk cluster degrades node by node; this one degrades all at once. That is the single biggest item on this list, and it is an availability argument, not a performance one. - **Write-path latency.** Reaching remote storage is slower than reaching an attached disk, and designs compensate with local staging, batching or more aggressive acknowledgement choices — each of which is its own trade. - **A different charge model.** Spend moves from provisioned disks to requests and stored bytes. Estates with very high small-operation rates can find the new bill surprising in either direction; it has to be modelled, not assumed. - **The migration.** Moving an existing estate across is a platform change per cluster, with a period of running both shapes. The saving is in future membership changes, so the payback depends entirely on how many of those you expect. - **Availability of the option.** Not every platform in this class offers the shape at all, and some offer it only for older records. The choice may be a platform choice in disguise. ## How to decide Ask, in order: 1. **How often does this estate change the size of a cluster?** Count the last year honestly. Twice is not a case; weekly is. 2. **What does a membership change cost today?** Hours of a named engineer, a scheduled slot, an elevated-risk period. Multiply by the count from step one. 3. **Can the estate tolerate a dependency shared by every node?** If a single shared store being slow means every cluster is slow at the same moment, does anything in the business notice differently than it would today? 4. **Does the write path have latency to give?** If the tightest stream on the estate is already close to its allowance, the compensations may cost more than the elasticity is worth. 5. **Is this one cluster or the estate?** The strongest version of this decision is usually narrow: move the clusters whose size genuinely moves, and leave the fixed ones where they are. A principal's deliverable here is not a preference. It is a stated position on which clusters are elastic, what the shared dependency is allowed to cost when it is unavailable, and what evidence would reverse the decision.

  • Does detached storage remove evacuation entirely?
    No. A leaving node still hands its units of ownership over, and it still has to settle anything it has not yet pushed to the shared store, so the completion condition survives. What collapses is the duration: seconds of handover instead of hours of copying. The discipline of confirming the node holds nothing is unchanged.
  • What is the strongest argument against the move?
    One store becomes a dependency of every node. A cluster on local disks degrades a node at a time and survives partial failure; a cluster whose records all live in one shared store degrades everywhere at once when that store is slow or unavailable. That trade is about availability, and it has to be accepted explicitly.
  • How do you judge the payback?
    Count membership changes over the last year and price each one in engineer hours, scheduled slots and elevated-risk time. Compare that to the write-path latency you give up, the new charge model and the migration itself. An estate that resizes twice a year has no case; one that follows a daily curve usually does.

saying these in an interview costs you the question

  • Presents detached storage as strictly better with no availability trade
  • Claims it removes evacuation rather than shortening it
  • Forgets that recently written records usually still live on the node
  • Assumes a handover means the new owner also has a warm cache
  • Argues the move for an estate that resizes clusters twice a year