Across a pipeline of services on different schema versions, how would you set one policy for unknown-field handling?
answer
- classify edges, not services
- authored input strict, transit preserving
- a fixture message from the future
- count unrecognised fields per hop
- preservation carries unvalidated data
basics
~20 sClassify edges rather than services: strict at boundaries where a human or tool authored the input, preserve-and-forward on internal transit hops, drop at terminal sinks. Then make the policy real with a conformance fixture, a per-hop counter, and a named owner.
solid answer
~50 sA policy that says "everyone should preserve unknown fields" is an aspiration until something enforces it. I would do four things. **Classify the edges** — authored input, internal transit, terminal sink — and assign a behaviour to each class, because the right answer differs by what the edge is for, not by which team owns the service. **Write the behaviour into each contract** with a named owner, so no reader inherits it from tooling defaults. **Make it testable**: every service ships a fixture message carrying a field its schema does not define, and asserts what it does with it. **Make it observable**: a counter of unrecognised fields per hop and per field identifier, so you learn when a hop is a version behind instead of discovering it from a customer. The trade to accept openly is that preservation makes every transit hop carry and store data it cannot validate.
go deeper
Understand that different services in one pipeline can behave differently toward fields they do not recognise, and that this is a decision someone should have made rather than an accident.
Be able to say what behaviour suits an authored-input edge versus an internal transit hop, and why a terminal sink can drop safely.
Show how you would enforce it: a fixture message carrying an undefined field in every reader's tests, plus a per-hop counter of unrecognised fields to catch drift early.
Set the rule others apply without asking, state preservation's cost in storage and unreviewed data, and decide whether the organisation relies on preservation or on fail-safe defaults.
## Why a per-service default is not a policy In a pipeline of a dozen services, unknown-field behaviour is usually the sum of a dozen unexamined tooling defaults. The result is not uniform and not knowable: a message survives one route and is quietly thinned on another, and which route it took depends on load balancing. A policy is what replaces that with a property you can state, test and observe. ## Classify edges, not services The correct behaviour depends on what the edge exists to do: | Edge class | Behaviour | Why | |---|---|---| | Authored input — configuration, policy documents, tool output | Strict rejection | The author is present and a discarded key is a silent failure they can fix now | | Internal transit — enrichers, routers, fan-out hops | Lenient, preserving | The hop is a courier; dropping here loses data the endpoints agreed on | | Terminal sink — stores its own projection, re-emits nothing | Lenient, dropping | Nothing is lost downstream because there is no downstream | | Vetting boundary — accepts from a less trusted zone | Strict, explicit about what it forwards | Forwarding unreviewed fields defeats the boundary's purpose | A service can sit on more than one edge — an ingest endpoint on one side, a transit hop on the other — so the policy attaches to interfaces. ## What preservation actually costs An honest policy states the price rather than pretending preservation is free: - **Carrying cost.** Every transit hop holds a second, untyped compartment beside its typed value, and pays for it in message size and memory. - **Storage cost.** A hop that persists messages now persists fields nobody at that hop can interpret or age out, which complicates retention and data-handling reviews. - **Blindness.** Preserved fields are forwarded without review. On an internal courier that is correct; across a trust boundary it is a way for unreviewed data to travel under a trusted service's name. - **Code discipline.** Preservation breaks the moment a service constructs a fresh outgoing value instead of editing the decoded one — the compartment is simply not copied, and no test that looks only at modelled fields will notice. ## Making the policy real 1. **Write it into the contract.** Each interface's contract states the unknown-field behaviour and names the owner who may change it. This is the step that stops a decoder setting from silently deciding the question. 2. **Ship a conformance fixture.** Every reader's test suite includes a "message from the future": valid bytes carrying a field the service's schema version does not define. A preserving service asserts the field appears in its output; a strict service asserts the rejection. 3. **Instrument every hop.** Count unrecognised fields by field identifier. The counter answers two operational questions at once — which hop is behind, and which newly added field is not yet understood everywhere. 4. **Gate schema review on the failure question.** Every added field records what a reader does when it never arrives, so defaults and unknown-field policy are decided together rather than separately. ## Where defaults meet the policy The two halves cover for each other, and the policy should say which half it is leaning on. A pipeline that preserves can tolerate a default whose loss would be harmful, because the value survives the hop. A pipeline that drops must insist that every default is fail-safe, since any field may be thinned in transit. A team that has decided neither is relying on a property no one has written down — which is exactly the state most pipelines are in before someone asks this question in a design review. ## What you are trading away Against uniform preservation stands a simpler and legitimate strategy: **constrain the topology**. If a field only ever crosses hops that model it, nothing needs preserving. That buys a smaller, more auditable data flow at the price of coupling routing to schema knowledge, and it stops scaling once the number of producers and consumers grows independent of each other. The judgment a lead is being asked for is where on that line the organisation sits, stated as a rule others can apply without asking — not a preference expressed once in a design document and then contradicted by whatever each service's decoder happens to do.
- What single test proves a service honours a preserve-and-forward policy?A fixture carrying a field the service's schema version does not define, pushed through the real decode-edit-encode path, with the output asserted to still carry it. Anything weaker — comparing decoded values, or checking only modelled fields — passes even when the compartment of unknown fields was silently dropped.
- Why does the policy attach to interfaces rather than to services?Because one service commonly sits on two edges at once: strict on an ingest endpoint where a person authored the document, lenient and preserving on the internal stream it forwards to. Assigning a single behaviour per service forces one of those two edges into the wrong answer.
- How does preservation change what a trust boundary is worth?A preserving boundary forwards fields nobody at that boundary examined, under its own name. That is right for an internal courier and wrong for a service whose reason to exist is vetting what crosses it, so boundaries should be explicit about what they forward rather than inheriting the transit policy.
saying these in an interview costs you the question
- States a fleet-wide preference with nothing enforcing it
- Assigns one behaviour per service instead of per interface
- Presents preservation as free of storage and review cost
- Forgets that a fresh outgoing value silently drops the preserved compartment
- Decides unknown-field policy without deciding default values
- Relies on every team reading a design document rather than on a test