A pre-trained compression dictionary would shrink millions of tiny messages that never fill a sliding window — what does adopting one commit both ends to?

## Why tiny messages need it A sliding-window matcher can only reference what is already behind it. A 200-byte message compressed on its own has nothing behind its first byte and very little behind its last, so almost everything goes out as literals: the field names, the punctuation, the enumerated values that every message in the path shares. The redundancy is real, but it is *between* messages, and an independently compressed message cannot reach it. An **explicit pre-trained dictionary** removes that handicap by priming the window with a block of representative content before compression starts, as if those bytes had just been emitted. The first byte of the message can now match into it. Nothing else about the scheme changes: the same match stage, the same minimum match length, the same entropy stage afterwards. The dictionary is not transmitted — which is the whole point, and also the source of every obligation that follows. ## The commitments, in order of how badly they bite 1. **Byte-identical shared state.** Distances resolve against the primed window, so a reader holding different dictionary bytes copies different content. This is not a degraded ratio; it is wrong output. 2. **Identity carried with the data.** Every compressed message must name the dictionary it used, so a reader can refuse rather than guess. Without an identifier the mismatch may not fail loudly — it can decode to plausible-looking wrong bytes. 3. **Immutability.** A dictionary is never edited in place. Retraining produces a new dictionary with a new identity, and the old one stays alive for as long as anything compressed against it exists. 4. **Deployment ordering.** Readers must hold a dictionary before any writer emits data referencing it. That is a two-phase rollout, and it constrains the order in which independently deployed services can ship. 5. **Retention.** Archived data is bound to the dictionary it was written with. Deleting or losing that artifact makes the data unreadable, so it inherits the retention policy and the backup policy of the data itself. 6. **Drift.** The dictionary reflects the sample it was trained on. As payload shape changes, its value decays quietly — the data keeps decoding, the ratio just sags — so it needs to be measured, not assumed. ## What you gain against what you now own | Gain | Obligation it creates | |---|---| | Small messages reference shared boilerplate immediately | Both ends must hold the same bytes, forever, for that data | | No per-message cost for common structure | An identifier and a negotiation or pinning mechanism in the frame | | Ratio improves without changing the message format | A versioned artifact with its own build, rollout and retention | | Works with an unchanged decoder algorithm | Deploy ordering coupling between independently released components | ## Where it does not pay The benefit is inversely proportional to payload size. A large payload warms its own window within its first fraction, after which the primed content is a rounding error on the total — the shared boilerplate is already behind it, in its own history. So the honest rule is that a pre-trained dictionary is for many small messages that are similar to each other and are compressed independently, and for nothing else. It is also worth remembering that the dictionary occupies window space; on a large payload it displaces the payload's own recent history, which is more valuable. Two more situations argue against it even when the ratio gain is real: - **Independent deployment you do not control.** If readers are operated by someone else, or cannot be upgraded on your schedule, the ordering requirement is a liability rather than a checklist item. - **Long-lived archives.** Data that must be readable in a decade with minimal external dependencies is better off paying the extra bytes than acquiring a mandatory companion artifact. ## How to decide, and what to say The judgement worth articulating is that this trade converts **bytes on the wire into operational coupling**. That can be an excellent trade — for a high-volume path of small, homogeneous messages between components released together, the saving is large and the coupling is already present. It is a poor trade where the readers are independent, the payloads are large, or the data outlives the systems that wrote it. If you take it, the guardrails are unglamorous and non-negotiable: an identifier in every frame, an immutable versioned artifact distributed like code, readers deployed first, an integrity check over the decompressed result so a mismatch fails loudly, a measured ratio to detect drift, and a retention rule tying the dictionary's lifetime to the data's. The decision is not whether compression is worth it; it is whether you want another distributed artifact whose absence makes your data unreadable.