After late trip uploads restate a driver's 30-day aggregate, why must the feature row be versioned rather than overwritten in place?

## Why one value per key is not enough The natural shape for a feature table is one row per entity: driver `D-8821` has a `harsh_braking_per_100km` of 2.9, updated whenever the pipeline recomputes. It serves request-time reads perfectly well, and it is useless for assembling training data, because it answers only one question - `what is the value now?` - while every training row asks a different one: `what was the value at this row's quote instant?` Restatement is what turns that from a theoretical objection into a defect. A 30-day window closes, the pipeline computes 1.4, a quote is priced against it, and four days later a buffered trip lands and the same window recomputes to 2.9. Under in-place update, the 1.4 no longer exists anywhere, and there is no record that it ever did. ## The Type 2 shape Instead of updating, append. Each value becomes a version with its own interval of validity in **knowledge time**: - `valid_from` - the instant the store began serving this value; - `valid_to` - the instant a newer version superseded it, left open for the current version; - the **event window** the value summarises, kept as its own start and end. Two rules keep it honest. Use **half-open intervals** (`valid_from <= t < valid_to`), so exactly one version matches any instant and no timestamp matches two. And write a new version **only when the value actually changes**, so a driver who has stopped uploading stops generating rows. | | in-place update | versioned interval | |---|---|---| | answers `value now?` | yes | yes, the open version | | answers `value on 3 April?` | no | yes | | training set reproducible | no, drifts with every rebuild | yes, identical forever | | storage | one row per driver | one row per change | | read cost at request time | a single lookup | a lookup plus an interval filter | | explains a past decision | no | yes, with the figures used | ## Reading it back An as-of join now has a predicate that means something: for each quote row, take the version where `valid_from <= quoted_at` and `valid_to` is either null or strictly after `quoted_at`. A March quote reads 1.4 forever, whatever the pipeline learns later; a quote after the restatement reads 2.9. Two rows for the same driver, days apart, legitimately disagree - which is the point. ## What it costs 1. **Storage grows with change, not with time.** Active drivers in a volatile window generate versions daily; dormant drivers generate none. Sizing follows the change rate, not the entity count. 2. **Every read gets an extra predicate.** The offline assembly pays an interval filter it did not pay before, which is a real but modest scan cost. 3. **Closing a version correctly is fiddly.** The old version's `valid_to` and the new version's `valid_from` must be the same instant, in the same clock, written in the same transaction - otherwise a gap or an overlap appears, and the join silently returns nothing or two rows. 4. **Old versions eventually need a policy.** Nothing here says keep them forever; it says keep them at least as far back as the oldest training row you intend to rebuild, and a conscious retention decision has to exist. ## The failure it prevents Without versioning, two training sets assembled from the same code a month apart disagree, and neither is wrong by its own lights. A model cannot be rebuilt from its recorded inputs, an offline comparison between a champion and a candidate quietly uses different data, and no one can reconstruct the figures that justified a particular price to a driver who asks. ## The part that is easy to get backwards The restated value is not a corruption. Once the buffered trip lands, 2.9 is a better description of that driver's 30 days than 1.4 ever was - and it is still not the number the March quote was priced on. Both facts are true at once, and versioning is precisely the mechanism that lets the store hold them both instead of choosing.