How do you diagnose a json.JSONDecodeError in a production ingest path?

## What the exception actually gives you `json.JSONDecodeError` lives in `json.decoder`, is re-exported as `json.JSONDecodeError`, and subclasses `ValueError`. That inheritance is a compatibility decision from when the decoder raised a bare `ValueError`, and it has a practical consequence: `except ValueError` works, but it is a blunt instrument that also catches an `int()` conversion three lines away. Catch `json.JSONDecodeError` by name. The instance carries four attributes worth knowing cold: - `msg` — what the parser expected at the failure point, e.g. `Expecting property name enclosed in double quotes`. - `doc` — the entire document string it was parsing. - `pos` — the zero-based character offset where parsing stopped. - `lineno` and `colno` — one-based line and column, computed from `pos`. `str(exc)` is `msg` plus the position, which is why the default traceback is already reasonably informative. What it does *not* tell you is what the surrounding text looked like — and in an ingest path you no longer have the payload in your hand. ## Logging the failure without leaking the payload The reflex fix — log the whole document so someone can look at it — is wrong twice over. Payloads are unbounded (a truncated multi-megabyte upload floods the log), and they routinely carry tokens, addresses or personal data that your logs are not cleared to hold. Slice a window instead. ```python import json try: record = json.loads(raw) except json.JSONDecodeError as exc: window = exc.doc[max(0, exc.pos - 40):exc.pos + 40] logger.warning( "malformed payload at line %d col %d: %s | near: %r", exc.lineno, exc.colno, exc.msg, window, ) ``` Even the window may need redaction depending on what flows through the endpoint; the size and the redaction policy are decisions, not defaults. Pair the log with a counter keyed on `msg`, because a spike of one identical message means a broken producer while a scatter of different ones means corruption in transit. ## Three failures that are not the same failure An ingest path — say the inbound webhook of an email-digest sender, run by a four-person team, that receives one delivery-report document per send — sees three distinct failure classes, and conflating them is the usual reason an incident takes hours instead of minutes. 1. **Not decodable as text.** If you hand `json.loads` raw `bytes` and they are not valid UTF-8 (or UTF-16/32, which it autodetects), you get `UnicodeDecodeError`, not `JSONDecodeError`. That is a transport or content-encoding bug — a mislabelled charset, a truncated gzip stream — and it has different owners and a different fix. 2. **Not valid JSON.** `JSONDecodeError`. Common real causes are a truncated body from a dropped connection (`Expecting ',' delimiter` or `Unterminated string` right at the end, with `pos` near `len(doc)`), an HTML error page delivered with a 200 (`Expecting value` at `pos` 0), or a producer emitting a Python `repr` with single quotes. 3. **Valid JSON, wrong shape.** It parses, then blows up downstream as `KeyError` or `TypeError`, or worse, silently defaults. This is not a decode problem at all, and treating it as one sends you looking at the parser instead of at the producer's contract. The `pos` value discriminates the second class quickly: `pos == 0` means the body was never JSON, `pos` at the very end means it was cut off. ## Handling, not just diagnosing At the boundary, decode failures are expected traffic, not exceptional events. The shape that holds up: - Catch `json.JSONDecodeError` around the parse **and nothing else** — keep the `try` to the one call, so a genuine bug in the handler below is not misreported as bad input. - Reject with a specific error to the producer where the protocol allows it, and record the raw body to a dead-letter store with a retention limit rather than to the log. - Never retry a parse failure. Malformed text is deterministic; a retry loop on `JSONDecodeError` just multiplies the load. - Do not assert on `exc.msg` text in tests. CPython keeps refining the decoder's wording between releases, and a test that pins the string is a test that breaks on upgrade. Assert on the exception type and on `pos`/`lineno` if you must. - Guard size **before** parsing. `json.loads` builds the whole graph in memory, so a hostile or runaway payload is a memory problem the exception handler never gets to see. ## One more decode-side setting `json.loads` accepts the bare tokens `NaN`, `Infinity` and `-Infinity` by default — a CPython extension. If a producer's numbers must be strictly conformant, pass `parse_constant=` a callable that raises, so you find out at the boundary instead of when a `float('nan')` propagates into an average downstream and turns every aggregate into `nan`.