skip to content

How do you build a canonical message so an HMAC verifies on both sides?

level: seniorimportance: should knowfreq 35%

answer

  1. Signing bytes is signing a spec
  2. Two messages, one rendering
  3. One message, two renderings
  4. Prefixing lengths beats trusting separators
  5. Never sign a parsed object

basics

~20 s

Define one exact byte encoding both sides implement: fixed field order, an unambiguous framing such as length prefixes, a pinned text encoding, and fixed formatting for numbers and timestamps. Sign those bytes, never a language object.

solid answer

~50 s

HMAC signs bytes, so a signing scheme is really a serialization spec. Write it down: which fields are covered, in what order, joined how, and with what encoding. The two failure modes are **ambiguity** and **drift**. Ambiguity is when two different field sets produce identical bytes — joining values with a separator that can appear inside a value lets an attacker shift the boundary; length-prefixing each field, or escaping the separator, removes it. Drift is when both sides intend the same message but render it differently: a float formatted as `0.30000000000000004` on one side and `0.30` on the other, a timestamp in a different form, a non-ASCII name encoded differently. Pin numbers to an exact format or carry them as integers, pin text to UTF-8, and put a scheme version and key identifier in the signed bytes so you can change the rules later.

code

python · 11 lines
python
def naive(fields: dict[str, str]) -> bytes:
    order = ("to", "list")
    return "|".join(fields[k] for k in order if fields[k]).encode("utf-8")


honest = {"to": "[email protected]", "list": "weekly"}
forged = {"to": "[email protected]|weekly", "list": ""}

print(naive(honest))
print(naive(forged))
print(naive(honest) == naive(forged))

go deeper

for a junior

Recall that HMAC signs bytes, not objects, and that both sides must produce byte-for-byte identical input. Encoding a string differently, or formatting a number differently, is enough to make a correct key look like a wrong one.

for a middle

Explain the two failure modes: two different messages rendering to the same bytes, and one message rendering two different ways. Be able to name concrete causes — a separator inside a value, an omitted empty field, a float formatted with str on one side.

for a senior

Show that you would write the canonical form down as a spec: fixed field order, length-prefixed framing, UTF-8, integer-formatted quantities, and a version marker. Recognise a rounding drift in a signed numeric field as a signing-scheme defect rather than a key problem.

for a principal

Own the scheme's lifecycle: how a version marker and key identifier let it evolve, which fields must be covered so the same payload is not replayable across endpoints or environments, and whether integrations get one shared implementation or one per team.

## Signing is a serialization contract `hmac.new(key, msg, "sha256")` takes bytes. It has no opinion about what those bytes mean, and it will happily produce a tag over any rendering of your data. Whether verification succeeds therefore depends entirely on whether the signer and the verifier produce the *identical* byte string from the *same* logical message. That contract — which fields, in which order, joined how — is the canonical form, and writing it down explicitly is most of the work. Two distinct things go wrong. ## Failure one: ambiguity Ambiguity is a security bug. It exists when two different logical messages canonicalise to the same bytes. The classic instance is joining fields with a separator that can also appear inside a field's value. Consider an email-digest sender signing a per-recipient message built as `to + "|" + list`. A recipient address of `[email protected]|weekly` whose list field is empty — and therefore dropped from the rendering — produces exactly the same bytes as the address `[email protected]` with the list `weekly`. A tag valid for one is valid for the other, and if any field is attacker-controlled, the attacker chooses which message you verify. There are two clean fixes. **Length-prefix** each field: emit the byte length and then the bytes, so no value can impersonate a boundary regardless of its content. Or **escape** the separator inside values, which works but is easy to get subtly wrong and needs the escaping to be part of the written spec. Length prefixing is the one to prefer, because it fails loudly rather than silently and needs no per-value inspection. A related instance is the *absent* field. If an optional field is simply omitted when empty, then the message with the field set to an empty string and the message without the field at all may render identically. Emit every covered field, always, with an explicit empty encoding. ## Failure two: drift Drift is an availability bug, and it is the one that actually pages people. Both sides believe they are signing the same message, but their renderings differ by a byte. **Floating point is the sharpest edge.** A per-recipient open rate computed as `0.1 + 0.2` is not `0.3`; it is `0.30000000000000004`, and Python's `str` of it says so. If the signer renders it with `str` and the verifier with a two-decimal format, or one side happens to round before signing, the tags differ and every delivery is rejected — a rounding drift that looks exactly like a wrong key. The fix is not to round consistently and hope: it is to keep money and quantities as integers of the smallest unit, and where a float genuinely must be signed, pin an exact format string in the spec so there is only one rendering. The same discipline applies elsewhere. **Timestamps** must have one form — integer seconds since the epoch as ASCII digits is the usual choice, because every date-time rendering has variants. **Text** must have one encoding, UTF-8, applied to a specified Unicode normalisation form if names or addresses can carry composed characters. **Booleans, nulls and empty collections** each need one spelling. If a case-insensitive field exists, lowercase it in the spec rather than hoping both sides do. ## Do not sign a parsed structure The corollary of all this is that you should not sign an object graph. Re-serializing a mapping is a canonicalisation, and if you did not specify it exactly, whichever library you used specified it for you — including key ordering and whitespace, which vary between implementations. Either sign the bytes as they were transmitted, or build the canonical bytes yourself from named fields under a written spec. Signing `str(some_dict)` is the version of this mistake that survives longest in a codebase, because it works right up until an insertion order changes. ## Make the scheme changeable A canonical form is a contract you will eventually want to change — a new covered field, a different hash. Put a short **version marker** in the signed bytes and reject unknown versions. Put a **key identifier** alongside the tag so the verifier can select the right secret without trial decryption, and so rotation does not depend on trying every key. Put the **algorithm** in the signed bytes rather than reading it from an attacker-supplied header, so nobody can talk you into a weaker one. Finally, cover everything security-relevant, not just the payload. If the same signed body is meaningful at two endpoints, or in two environments, include the target in the signed message. A tag proves the bytes came from a key holder; it only proves *what they meant* if the meaning is inside the bytes. ## The test to apply Before shipping a scheme, ask one question of it: can any two distinct logical messages produce the same canonical bytes, and can any one logical message produce two different canonical bytes? The first is the forgery hole; the second is the outage. A length-prefixed, fixed-order, fixed-encoding, version-tagged rendering answers no to both.

  • Why is length-prefixing each field preferable to escaping the separator?
    Because it removes the ambiguity structurally rather than by inspection. With a length prefix, a field's contents can never be mistaken for a boundary no matter what bytes it holds, and the reader knows exactly how far to read. Escaping works, but it depends on both sides implementing identical escape and unescape rules for every edge case — a nested separator, a trailing escape character, a multi-byte character — and a single asymmetry between the implementations reintroduces the hole.
  • Why include a scheme version and a key identifier in the signed bytes?
    The version lets you change what is covered later: the verifier reads it, rejects unknown versions, and can accept two forms during a migration without guessing. The key identifier lets the verifier pick the right secret directly instead of trying each one, which matters when a rotation set grows. Both belong inside the signed bytes rather than only in a header, so an attacker cannot downgrade the scheme or point verification at a different key.
  • A team wants to sign a Python dict directly since insertion order is guaranteed. What is wrong with that?
    Guaranteed iteration order says the mapping preserves the order it was built in — it does not say the two sides built it in the same order. One service constructing the mapping from a database row and another from a parsed request will iterate differently, and the bytes diverge. Beyond order, `str` of a mapping is a Python-specific rendering with its own quoting and float formatting that no other language reproduces. Build the canonical bytes from named fields under a written spec instead.

saying these in an interview costs you the question

  • Joining fields with a separator that can appear in a value
  • Signing str() of a dict and relying on insertion order
  • Omitting empty optional fields from the signed message
  • Signing a float using whatever str() happens to produce
  • No version marker, so the scheme can never change
  • Reading the algorithm from an attacker-supplied header

context

resolving links…