How do you govern a structured log-record schema across services, and what does it cost?

## What you are actually deciding Once logs are parsed by machines, the record shape stops being a formatting preference and becomes a published interface. A dashboard, an alert rule, a retention policy and an on-call runbook all bind to field names. Renaming `job_id` to `jobId` is then an API break with no compiler to catch it. So the governance question — what is in the record, who may add to it, and how a change ships — matters more than the choice of serializer. ## A small core plus a namespaced extension area The core is the set of fields every consumer may assume: a timestamp, a level, the logger name, the message text, a correlation id, the service name and its deployed version. Keep it short; a field in the core is a promise every service must keep forever. Everything else goes under one namespaced key — an `app` or `ctx` object — rather than at the top level. This buys two things. It stops teams colliding with each other, and it protects you from CPython itself: keys passed via `extra` are written straight onto the `LogRecord`, and `Logger.makeRecord` raises `KeyError` when the name is already taken. The set of taken names grows between releases — `taskName` was added in 3.12 and immediately reserved that word — so a top-level field name is a bet on future interpreter versions that a namespaced one does not make. Interpreter upgrades should not be able to turn a log call into an exception. ## Determinism beats prettiness A log record read by a machine wants values it cannot misread. Emit the timestamp as `record.created`, an epoch float: no timezone, no locale, sortable, exact. If you also want human-readable text, pin it — `Formatter.converter = time.gmtime` with an explicit ISO-like `default_time_format` — because `Formatter.formatTime` otherwise runs `time.localtime` through `time.strftime`, and codes such as `%c` or `%x` render according to the host's locale settings. That is not theoretical. A nightly six-hour thumbnail batch is the kind of job that gets moved onto a spare host, and if the format string was left locale-dependent the same code emits a differently-shaped timestamp there — so the pipeline's parser rejects the batch's records, and you discover it when you go looking for the run that failed and find no logs at all. Any field a human might localize — timestamps, decimal separators, month names, sorted enum labels — belongs in an invariant machine form, with localization left to whatever displays it. ## Price it Structured logging is not free and a lead should be able to say what it costs. - **CPU**: serialization runs once per handler per record, on the calling thread. Records that never pass a level check cost nothing, which makes level discipline the biggest lever you have; where building the *arguments* is expensive, `logger.isEnabledFor` is the guard. - **Bytes**: JSON is several times the size of the equivalent line, and downstream storage and ingest are usually billed by volume. Compact separators, short key names and dropping empty fields are real savings at scale. - **Cardinality**: a field with unbounded distinct values is cheap to emit and expensive to index. Ids belong in fields; free-form user text does not. - **Humans**: JSON is miserable to read in a terminal. Keep a plain-text handler for local development and interactive debugging — formatters are per handler precisely so both can coexist. ## Safety rules that belong in the contract Structured fields are indexed, retained and readable by more people than the service's own team, so a secret in a field is worse than the same secret in prose. State the rule where the schema is defined: no credentials, tokens or personal data as fields; log identifiers rather than values; cap field sizes; prefer an explicit allow-list of emitted fields over dumping the record's whole attribute dictionary, so a hasty `extra={"user": user}` cannot export an object's entire repr into a searchable store. Where user-controlled text does get logged, it is data, not markup — serializing through `json.dumps` escapes it, which is one more reason to build the payload as a dict rather than concatenating strings. ## Ownership and change Give the contract a home: one internal formatter and filter, wired the same way in every service, with tests that assert the core fields exist and parse. Then changes have a place to happen. Additions are safe and go first; a removal or rename runs on a clock — emit both shapes, migrate the consumers, then drop the old one — and a `schema` version field makes it possible for a consumer to know which shape it has. This is the same discipline as any other published interface, and the fact that a log line looks informal is exactly why teams forget to apply it. ## When not to do it A command-line tool, a one-off script and a local development run are all better served by readable text. Structured records earn their cost when something other than a human reads them; if nothing does, JSON is ceremony that makes debugging harder.