When a stream uses remote offload, which segments leave the data volume, and what does that change about writes?

## What remote offload actually moves **Remote offload** is the arrangement where a broker or streaming platform stops keeping all of a stream's readable history on the node's **data volume** and instead places **closed segments** in a remote object store. A **segment** is the unit the store rolls shut once it stops being written to; the one currently being appended to is **the active segment**. The unit matters. Offload operates on whole segments, never on individual records, because a closed segment no longer changes and is self-contained, which is what makes it safe to copy elsewhere and then drop locally. Two things stay behind: - **The active segment.** It is still being written, so there is nothing stable to upload. No design moves it. - **However much recent history the implementation keeps local.** Most keep a span of recent closed segments on the device as well, so that a reader working anywhere near the newest records never leaves the node. Designs differ in how that span is expressed - by age, by bytes, or not exposed at all on a rented platform - but the intent is the same: the front of the history stays where it is fast. The position where locally held segments end and offloaded ones begin is **the local boundary**. It is the single most consequential thing offload introduces, and it is the subject of everything that goes wrong later. ## What changes and what does not | Property | Before offload | After offload | |---|---|---| | Cap on how much history you may keep | the size of the data volume | the remote store's capacity and the bill | | Write path | local append | unchanged | | Reads of recent history | local | local | | Reads of older history | local | a fetch from the remote object store | | Things that must work for history to be readable | the cluster | the cluster, the remote store, and the metadata locating each offloaded copy | The first row is the entire point. **The retention window** - the span of history still readable on the store right now - is **the replay budget**: it is how far back a reader may be sent, how long a broken downstream may stay broken before its inputs are gone, and how much of an incident you may reconstruct afterwards. Before offload, that budget is negotiated against the size of a block device and against every other stream sharing it. After offload, it is negotiated against a storage bill instead, which is a far easier argument to win and a far easier one to lose track of. ## Capacity, not speed The claim to hold onto is that offload is a **capacity** mechanism and never a **performance** one: - Writes are unaffected, because they go to the active segment, which never left. - Reads of recent records are unaffected, because those segments are still local. - Reads of older records get **slower and separately charged**, because they now cross a network to a remote object store that charges for retrieval. - Nothing about offload makes a reader that cannot keep up go faster. If anything it does the opposite, because the reader furthest behind is the one that ends up on the remote path. So a team that enables offload to relieve a throughput problem has bought nothing, and a team that enables it to stop capping history at what one device holds has bought exactly what it wanted. ## Checking a capacity claim before you believe it 1. **Confirm the bound that actually binds.** If history is being cut by a configured age bound rather than by the size of the device, offload changes nothing until that bound is raised too. 2. **Account for what is still local.** The active segment, the locally retained recent span, and any segment still waiting to be uploaded all occupy the device. Upload is asynchronous on essentially every design, so a segment briefly exists in both places. 3. **Decide where the local boundary sits before turning it on.** Size the local span from how far back you expect ordinary reading to reach, not from whatever the default happens to be. ## Where designs genuinely differ - **Whether it exists at all.** A platform that keeps a replayable history of segments has something to move. A destructive-read design that removes a record once it is acknowledged has little history to relocate, though some such platforms can spill an oversized store of undelivered records to remote storage. - **Who does the remote read.** On some platforms the broker fetches the remote bytes and serves them to the reader, so clients need no change but the node pays. On others the reader is handed a location and fetches directly, which moves the cost off the node and puts it on the client. - **Whether retrieved bytes are cached locally.** Some implementations stage a retrieved range on the device; some do not, so a second pass over the same history is fetched, and charged, again. - **How much control you have.** Where the boundary sits is an operator setting on a self-run cluster and may not be exposed at all on a rented one.