Two producers write the same logical record, one omitting a field and one sending its default; why do their content ids differ?

## Why both writers are legal An encoding that supports optional fields generally accepts both documents: the field omitted, and the field present carrying the value a reader would have supplied anyway. A reader that fills defaults treats them as the same value — and that is exactly why the divergence is so confusing when it reaches a content-addressed store. The id is not computed from what the reader concluded; it is computed from the octets the writer produced. Two legal spellings, two byte strings, two ids, one logical record. The practical symptoms are unglamorous and expensive: a dedup table that stores the same artefact twice, a cache that misses for one producer and hits for another, an idempotency key that fails to suppress a retry issued by a different service, and a diff tool that reports two artefacts as distinct while showing no differing field. ## Picking a presence rule A canonical profile must decree one rule. There are two, and the choice has consequences well beyond bytes: | Rule | What the canonicaliser does | What you gain | What you pay | |---|---|---|---| | Omit at default | Drops every field whose value equals its declared default | Ids do not move when a defaulted field is added, as long as writers leave it unset | "Absent" and "present at the default" collapse onto one id | | Always emit | Writes every field in the schema, filling unset ones from their defaults | The encoded form is self-contained and uniform | The id is a function of the schema version; recomputed ids move when the schema grows | Walk the always-emit case explicitly, because candidates get its direction backwards. A new optional field with a default is added. Nothing about the stored octets of an old record changes. But the moment anyone **recomputes** that record's id — a re-upload, a migration, a rebuild from storage — the canonicaliser now emits the new field too, the bytes are longer, and the id is different. Under omit-at-default, the same recomputation leaves the bytes untouched while the field is unset, so the id survives. That asymmetry is the whole reason to choose deliberately rather than by accident. ## The distinction you may be erasing Omit-at-default is the safer default *only* when your contract does not distinguish absence from the default value. Where it does — a partial update whose omitted fields mean "leave alone", a measurement whose absence means "not taken" rather than zero — an omit-at-default canonicaliser destroys information before the digest is computed, and two records with genuinely different meanings receive one id. The fix is not to change the presence rule; it is to stop leaning on defaults for a distinction that matters. Model the difference **in the value**: a field whose type can hold "not supplied" as a value of its own, so that presence stops being a spelling decision and becomes part of what is being encoded. ## Designing it out 1. **Normalise the value before the canonical encoder sees it.** Resolve which fields are semantically present, then hand the encoder a value with no remaining ambiguity. 2. **Prefer shapes with no defaults** for anything that is content-addressed. A schema whose fields are all explicitly carried has no presence freedom left to pin. 3. **Record the profile version next to the digest.** Any change to the presence rule changes every recomputed id, so a verifier needs to know which rules produced the id it is holding. 4. **Test the pair explicitly.** Keep a fixture of the same record in both spellings, and assert that canonicalising both yields one byte sequence. ## What to check in review - Does any producer compute the id from its own writer's output rather than from the canonical encoder? That is the usual root cause. - Does a hop fill defaults on the way through, so a record that arrived sparse leaves dense? - Does the contract anywhere give absence a meaning of its own, while the profile omits defaults? - If the schema gained a defaulted field last quarter, did any ids move — and did anything depend on them not moving?