Why can verifying a webhook signature after decoding the body to str fail?

## What actually gets signed A webhook signature is an HMAC over the request body as a **sequence of octets**. The sender hashed exactly the octets it put on the wire; nothing about characters, Unicode or your codec choice enters into it. Your receiver must hash exactly the octets it took off the wire. That is the whole content of the bug. `bytes` -> `str` -> `bytes` is not an identity function in practice, even when both steps use the same codec, because the middle value is *text* and text has no memory of how it was spelled. Everything the round trip loses is a place a digest can diverge: - A byte-order mark at the start of the payload is decoded into a code point and often stripped or re-emitted differently. - A layer that parses the JSON and re-serialises it changes key order, whitespace and escaping. The object is equal; the octets are not. - A helper that formats or logs the body through a locale-dependent representation, then passes the reformatted text on, replaces separators and quoting. - The producer sent text that is valid in its encoding but not in the one you assumed, so decoding raises before the check runs at all. The symptom is characteristic: verification passes for most traffic and fails for a minority. The failing minority is precisely the payloads containing something non-ASCII, which are also the payloads your fixtures never contain. On an 11-person team the first three engineers to look at it will each conclude the shared secret is out of sync, because that is the failure mode everyone has seen before. ## Why the transport hands you bytes in the first place Sockets and binary-mode files return `bytes` because octets are what the medium carries and stores. `socket.socket.recv` cannot know the peer's encoding — the wire format does not carry it inside the payload. At best it is declared out of band, in a `Content-Type` charset parameter, and that declaration is a claim by the sender rather than a fact about the octets. Framework request objects therefore keep a raw body available, and it is that value, not the parsed or decoded one, that a signature check must consume. ## The boundary rule ```python import hashlib import hmac import json def handle(raw: bytes, header_sig: str, secret: bytes) -> dict: expected = hmac.new(secret, raw, hashlib.sha256).hexdigest() if not hmac.compare_digest(expected, header_sig): raise ValueError("bad signature") return json.loads(raw) # json.loads accepts bytes directly ``` Three things are load-bearing. First, `raw` is the untouched body; capture it before any middleware can normalise it, and if a framework only exposes a stream, read it once and keep it. Second, `hmac.compare_digest` compares in constant time so the comparison itself does not leak. Third, the decode either does not happen at all — `json.loads` takes `bytes` and applies the JSON specification's own UTF-8 rule — or it happens *after* verification, so malformed input becomes a clean rejection rather than an exception on an unverified payload. ## Ordering matters as much as the type Decoding before verifying is a second, subtler defect. An attacker or a broken producer can send octets that are not valid in your assumed codec; if the decode runs first, the request dies in a codec exception, on a path where you have not yet established the payload is even from a legitimate sender. Verifying first means unauthenticated traffic is rejected cheaply and every codec error you do see comes from a sender who holds the secret, which makes the error actionable rather than noise. ## Diagnosing an existing receiver When a receiver is already failing intermittently, do not start with the secret. Log the length of the body you hashed alongside the length the sender reports, and the first bytes of each. A length mismatch on exactly the non-ASCII requests is conclusive: something between the socket and your hash re-encoded the payload. Then walk the path from the framework's raw body to your check and find the layer that produced text. It is usually a helper written for a legitimate reason — logging, validation, size limits — that returned its reformatted value instead of the original. ## What an interviewer is checking That you understand a digest is a function of octets, that you know where the `bytes`/`str` boundary sits in a web application, and that you can order the operations correctly under adversarial input. It is a fine proxy for whether someone has actually operated a receiver rather than only written one.