A content-addressed store keys artefacts by a digest of their encoded bytes, so why can two services encoding the same value produce different ids?

## Why one value has many byte spellings An encoder turns a value into bytes; a published encoding defines which byte sequences are **valid** and what each one means. Those are not the same job. Almost every general-purpose encoding specifies a grammar and leaves the writer free wherever that grammar accepts more than one spelling of the same value. The freedoms that actually bite in production: - **Member order.** In an encoding whose objects or maps carry member names, the grammar accepts the members in any order. A writer that iterates a hash-based map can emit two different orders in two runs of the *same* build. - **Insignificant whitespace.** Indentation, a space after a separator, a trailing newline: meaning-preserving, byte-changing. - **Number spelling.** `1`, `1.0`, `1e0` and `0001` may all denote one value in a text encoding; a binary encoding commonly offers both a one-byte and a wider spelling of a small integer. Under IEEE 754, negative zero and the many bit patterns that all mean "not a number" are the same trap in binary form. - **String escapes.** A character may appear literally or as an escape; escape hex digits may be upper or lower case; only some characters *must* be escaped. - **Text normalization.** The same visible text has more than one Unicode code-point sequence (NFC against NFD), and UTF-8 turns those into different bytes. - **Presence at default.** A field omitted and the same field written carrying the value a reader would have defaulted it to usually mean the same thing to that reader, and are different bytes. - **Non-minimal framing.** Binary encodings often permit a length or tag written wider than it needs to be. ## What a canonical encoding actually is A **canonical** — or *deterministic* — encoding is not a separate format. It is a **profile over an encoding that removes each freedom by decreeing exactly one choice**, so that `encode` stops being a relation and becomes a **function** from value to bytes. Once it is a function, `digest(encode(value))` is a function of the value, which is the property a content id, a dedup key or a byte-level check actually depends on. A usable profile pins at least these: | Freedom | What the profile must decree | |---|---| | Member order | One total order — typically byte-wise over the encoded member name, never a locale-sensitive collation | | Whitespace | None at all, or one exact layout | | Integers | The shortest form that represents the value | | Floating point | One spelling per value; usually forbid the payloads that have no single spelling | | Strings | One Unicode normalization form and one escaping rule | | Presence | Either always omit a default-valued field, or always emit it | | Collections | An order for anything the domain treats as unordered | ## What it buys With the profile in force, the digest becomes a **name for the value**. That single property underwrites several everyday mechanisms at once: an artefact id in a content-addressed store, a dedup key that lets two uploads of one value share storage, a cache key that hits across machines, an idempotency key that survives a retry from a different process, and a byte-level integrity check whose expected digest can be recomputed rather than merely remembered. ## Two failure modes worth naming in the interview 1. **Encoding-equal is not domain-equal.** Canonical form makes the *representation* deterministic. It does not decide that `10.00` and `10` are the same money amount, that a list the domain treats as a set should be sorted, or that an identifier is case-insensitive. Those are decisions about the value, and they have to be made *before* encoding — usually by normalising the value itself, so that the canonical encoder is handed something already in a single form. 2. **"We sorted the keys, so we are canonical."** Ordering is the most visible freedom, not the only one. A system that sorts members but leaves number spelling, escaping, normalization and presence free still produces unstable ids; the instability just becomes rarer, which makes it harder to find. ## Where the promise comes from Encodings differ in how much of this they guarantee. Some publish a deterministic profile as part of the standard; others explicitly decline to promise that two implementations — or two versions of one implementation — emit identical bytes for the same value. Where the promise is missing, the profile becomes yours to write down, version and test, and the test that matters is cross-implementation: encode one corpus with every writer you support and compare digests, rather than asserting that a single writer agrees with itself.