Your archive stores each document as one adaptive arithmetic-coded stream — what durability risks does that create?

## What this design actually couples together A single adaptive arithmetic-coded stream is the most entangled representation a lossless pipeline can produce. Every design virtue it has — no table shipped, fractional-bit costs, probabilities that track the document — comes from the same property: **every bit depends on every preceding bit and on the exact state of a model that was never written down.** For a stream that lives for a decade, that property has to be managed deliberately, because nothing in the coder manages it for you. ## Risk 1: the decoder is the format With a static model you can store the table beside the data and a future reader has what it needs. With an adaptive model there is no table — only a **rule**. Reconstructing the document years later requires reproducing exactly: - the initial counts and the alphabet's order along the line; - the increment applied after each symbol; - the threshold at which counts are rescaled and how that halving rounds; - the register width and the integer arithmetic used for slice boundaries; - the escape or end-of-message convention. Get any one wrong and the output is wrong, not missing. So the practical decisions are: write that rule down as a specification rather than leaving it in one implementation; give it a **version number stored in every stream's header**; keep a reference decoder and a corpus of round-trip fixtures as archive artefacts in their own right; and forbid floating point anywhere in the slice computation so that a decoder built later on different hardware computes identical boundaries. The cheaper alternative is worth naming explicitly: a **frozen static model stored with the archive**. It compresses a little worse, but a reader only needs the table plus a far simpler coder, which is a real trade for very long horizons. ## Risk 2: no resynchronisation There are no codeword boundaries in the output, so there is nothing to resume from. A bit that flips in storage moves a slice boundary, the next symbol decodes differently, the model is then updated with that wrong symbol, and the divergence compounds. Two consequences worth stating plainly: 1. The corruption is **silent**. The decoder produces structurally valid symbols; without an independent check, bad output looks like good output. 2. The blast radius is **the rest of the unit**, not a page. ## Risk 3: read amplification Random access is impossible within a coded unit: reaching the last symbol means running the model over everything before it. For an archive whose access pattern is "fetch one document occasionally", that may be fine. For one that serves ranges out of large objects, it is a serious cost, and it is structural rather than a tuning parameter. ## The containment, and the trade it forces All three risks are bounded by the same decision — **how big is an independently coded block** — and that decision is the actual judgement call: | block size | compression | blast radius of corruption | cost to read a fragment | |---|---|---|---| | whole document | best; the model learns once | the whole document | decode everything | | moderate blocks | slightly worse; model relearns per block | one block | decode one block | | small blocks | noticeably worse; learning cost dominates | one small block | cheap | Around that choice sit four practices that are not optional: - **Reset both** the interval and the model state at each block boundary, and terminate each block explicitly with a reserved symbol or a stored length. A block that depends on the previous block's model has not actually been made independent. - **Checksum the decoded content**, not only the coded bytes, so a model or format mismatch is caught as well as bit rot. - **Verify on write** by decoding the stream back and comparing before the original is released — the one check that catches an encoder-decoder asymmetry before it is archived. - **Put repair outside the coder.** An interval coder has no error tolerance of its own; whatever redundancy the archive needs belongs to a layer above it, and that layer's unit should line up with the block boundaries so a repair restores something independently decodable. ## How to frame the answer The judgement being probed is not "which coder". It is recognising that choosing a maximally adaptive representation moves risk from the storage layer into the **format definition** and into the **blast radius of a single fault**, and then pricing that against the compression it buys. A lead who reaches for block framing, an explicit versioned model specification and verified writes has understood the trade; one who answers only with a compression ratio has not.