A new per-utterance field and a changed audio sample rate land mid-corpus - what keeps last quarter's snapshot readable?

## The schema belongs to the snapshot, not to the code The rule that makes old snapshots readable is a single inversion: the reader does not apply *its* schema to the data, it resolves the schema **the snapshot declares**. The manifest records a schema version, the schema registry holds every version ever published, and a reader loading a two-year-old snapshot interprets it exactly as it was written. The alternative - one current schema that all data must conform to - forces either backfills or breakage, and backfilling a frozen snapshot is the one thing versioning exists to prevent. ## Three kinds of change, only two of which a checker can see | change | mechanically detectable | what old snapshots need | |---|---|---| | add an optional field | yes, trivially | nothing; readers see an explicit absence | | rename, remove or re-type a field | yes | a new major schema version; old snapshots stay on the old one | | same name and type, different meaning | **no** | a new field or a declared per-shard property, plus a recorded boundary | The third row is the dangerous one, and the sample-rate change is its textbook instance. The field still holds audio, still has the same type, still passes every compatibility gate - and the model now sees two populations under one name. ## Missing is not zero When a reader meets a snapshot written before a field existed, it has three options and only one of them is honest: - **Represent it as absent** and let the trainer decide - exclude those rows, carry an explicit missingness indicator, or use a model that handles absence. Honest. - **Default it to a neutral-looking value** such as zero or an empty string. Dishonest: the trainer cannot distinguish 'no value existed' from 'the value was zero', and it learns the difference as signal. - **Backfill it into the old shards.** Worst of the three: it changes the content behind published hashes, so either the ids stop resolving or, if the ids are re-pointed, every past claim about what trained a model becomes false. ## Catching an undeclared change A change of meaning is invisible to schema comparison, so the defence is **statistical and written at ingest**: record per-shard properties - sample rate, channel count, bit depth, duration distribution, the proportion of rows carrying each optional field - in the manifest at write time, and compare them across the shards of one snapshot and across consecutive snapshots. A property that shifts *inside* a single snapshot is the alarm, and it is only visible because the writer recorded it; nothing can be recovered later from bytes that never declared what they were. ## Derived values are versioned too Normalisation statistics, the vocabulary, the feature-extraction window - anything computed **from** a snapshot inherits the snapshot's version and must be recorded alongside it rather than recomputed from the current corpus at read time. Recomputing quietly imports today's distribution into an old snapshot's rebuild, which is the same defect as a backfill wearing different clothes. ## Failing loudly A reader that encounters a schema major it does not understand must **refuse**, not drop the fields it does not recognise. Silent field-dropping is how a training job quietly loses an input it was supposed to have and produces a worse model with no error anywhere. The corresponding rule on the write side is that a new schema version is published before the first shard that uses it, so no shard ever references a schema nobody can resolve. ## The design-round answer Schema version per snapshot, resolved from the snapshot; additive fields optional and absent rather than defaulted; semantic changes get a new name and a recorded boundary; nothing is ever backfilled into a frozen snapshot; per-shard statistics written at ingest are what catch what the schema check cannot see.