In structured logs, one service emits `attempt` as a number and another as a string. What breaks?

## Three shapes drift takes "Schema drift" in structured logs is not one problem. It is three, and they fail differently. 1. **Type drift.** `attempt` is a number in one service and the string `"3"` in another. Same name, same meaning, incompatible representation. 2. **Depth drift.** One service emits `user_id` at the top level; another nests it three deep under an object, so the queryable path is different even though nothing about the value changed. 3. **Presence drift.** A field is emitted on most lines and quietly omitted on some — an error path that returns before the value is known, a code path that never set it. All three come from the same root cause: a structured log line's shape is decided independently by every service that writes one, and nothing in the emitting process validates it against what anyone else is emitting. ## What a schema-on-write store does with each A store that fixes a field's type when it first sees it has no good options once a conflicting value arrives, because the earlier data is already committed under the earlier type. | Drift | Typical outcome | Who notices | |---|---|---| | Type conflict | the whole record is refused, or that field is dropped and the rest kept | nobody, unless someone watches the reject count | | Deep nesting | a distinct queryable path per depth; the number of distinct field names grows | whoever pays for the store, eventually | | Sometimes absent | the line simply does not match any filter on that field | whoever trusts a filtered count | Two of those three are silent by construction, and the third is silent for weeks. ## Why it is silent The failure happens on the far side of a one-way boundary. The emitting service handed a line to a local writer and got no error, because from its point of view the write succeeded — the loss occurs later, in a component that has no channel back to the code that produced the data. So there is no exception, no failed request, no retry, and nothing in the service's own health signals moves. What you see instead is a *hole*: a query returns fewer results than it should, a dashboard panel that used to have a line on it goes flat, a count that everyone trusts is quietly wrong. A hole looks exactly like good news. "That service stopped erroring" and "that service's error lines stopped being accepted" render identically. ## A worked case A school-meal ordering service runs a 320 ms p99 budget on order confirmation. A new region is brought up in a hurry with no instrumentation of its own, and its ordering service — built from a slightly older template — emits `attempt` as a string where the established regions emit it as a number. The store fixed `attempt` as numeric months ago. Roughly 12.4% of the new region's lines carry `attempt`, and every one of those records is refused. The dashboards do not break. They show the new region running comfortably inside budget, because the lines that carried retries — the slow ones, the ones with `attempt` above 1 — are precisely the ones being thrown away. The reported p99 is a percentile over a biased sample, and it is biased in the flattering direction. This is the reason type drift is worth an interview question at all: it does not degrade the data uniformly, it removes a correlated subset. ## Catching it before it costs you - **Own the field vocabulary in one place.** A shared logging helper or a small library that both names and types the common fields is the only intervention that scales past a handful of services; a written convention with no code behind it drifts within two quarters. - **Alert on rejects, not on volume alone.** The store knows how many records it refused and why. If nobody has wired that count to anything, drift is undetectable by design. - **Watch per-service line rates.** A service whose accepted-line count falls sharply while its request rate is flat is the signature of drift, and it is visible without touching the log content. - **Budget nesting explicitly.** Flatten to one or two levels and use a compound field name rather than depth; deep, free-form objects turn every distinct sub-key into another queryable field name. - **Decide the absent case per field.** Either the field is always present with an explicit "unknown" value, or it is genuinely optional and every query on it is written knowing that. What hurts is an optional field that everyone treats as mandatory. ## The mental model Structured logs are a schema you did not write down. Type drift is what happens when a schema exists implicitly, is enforced somewhere far from the code, and reports its violations to nobody. The fix is never "be more careful in code review" — it is to make the shape a shared artefact and to put a monitored signal on the enforcement point.