Feature Engineering & Feature Stores

Training and dispatch read the same feature with opposite access patterns: a training job scans months of one column across millions of rows, while dispatch fetches every feature for one driver in milliseconds. No single store serves both shapes well.

Event time is when the driving happened; ingestion time is when the trip summary reached the feature pipeline and became readable. Devices buffer and retry, so the two differ by hours or days and records arrive out of order.

A model validates nothing about what its inputs mean: any number in the right slot is scored. Training-serving skew therefore lands as a confidently wrong estimate rather than an exception, and shows up only when real outcomes arrive.

An embedding's coordinates only mean something relative to the encoder weights that produced them. The version stamp lets the serving path prove the vector came from the encoder the value model was trained against; without it, a mismatch is silent.

It commits the platform to serving that feature no older than the stated age, and to noticing when it cannot. The bound is a contract on the value a request reads, not a description of how often the job runs.

It means every value in the online tier can be reproduced from the columnar history by a materialization job. The online row is an output of the platform, not an input to it, so losing the tier is an availability incident rather than data loss.

A latest-value join hands every training row aggregates computed from trips driven after that quote was priced, so the model learns from facts no quoting system could have held. The offline score then measures hindsight rather than skill.

Three families: one definition implemented twice and diverging in units, rounding or window; a missing-entity default chosen differently on each path; and a serving read that returns a row older than the request it is answering.

The rolling refresh is mixing two coordinate systems in one population: re-encoded players now sit in a space the value model was never fitted to. Nothing errors, and the error grows in step with refresh coverage.

Log the exact input vector the model was scored on with the entity key, the serving timestamp and the definition version, then recompute each field from history for that same entity and instant and diff them field by field.

512 coordinates at four bytes each is 2,048 bytes per player, so eighty million players hold roughly 164 GB of vector payload before keys, stamps or extra copies. Halving the width to 256 coordinates halves that to about 82 GB.

The model meets a value its training column never held. Zero sits below every learnt prep-time split, so brand-new kitchens are scored like the fastest ones and the estimate comes out far too short for the cohort least able to absorb it.

A separate namespace keeps the two coordinate systems apart, so the switch is one pinned-version flip instead of a nightly-growing mixture, and the superseded vectors survive for rollback and for re-running old evaluations. The cost is double storage during the overlap.

Write the corrected values under a new definition name, rebuild history into it in re-runnable chunks, then move readers over once the new series is complete. Overwriting the old series in place destroys the record training was built on.

The bound is a ceiling on end-to-end delay, so you pick the cheapest mode that fits under it: batch for a daily bound, continuous computation for a sixty-second one, request-time for anything that depends on the request.

Three defensible responses: serve the prediction with the staleness recorded, substitute a coarser feature that is still inside its own bound, or refuse to score and let the caller fall back. Serving silently is the one wrong answer.

That shape issues 1,200 reads per request, and the request cannot finish until the slowest one returns. With a 1% per-read tail, almost every request contains at least one slow read, so request latency tracks the read tier's extreme tail rather than its median.

It receives nothing, and nothing is not an error: the read returns empty, the serving code fills the slot, and the model scores as if the filled value were real. The ranking degrades on every request with no failure anywhere to alert on.

Overwriting destroys what the feature store held at earlier quote instants, so every training row assembled afterwards silently receives the restated value. Storing each value with a valid-from and valid-to interval lets an as-of join reproduce what a past quote actually saw.

An event-time bound only limits which trips the value summarises; it says nothing about when that value was computed. A window recomputed after late uploads carries figures the quoting path never held, so the row still reads the future.

It removes implementation divergence - units, rounding, window bounds, filters - and the missing-entity default if that lives inside the artifact. It does not remove a stale input, an input the request path cannot fetch in budget, or a training job that quietly runs its own query.

A job's exit status says a run finished, not that homes hold fresh values. Age measured on what reads actually return catches a partial write, a stalled partition and a lagging source — all of which leave the job green.

Because the row count is the product of the two key cardinalities. Two million drivers and five thousand zones imply ten billion rows to write on every refresh, while a single request ever reads forty of them — the write side pays for a key space the read side never touches.

Without a lower bound, the join matches the newest version that exists, however ancient. A driver who stopped uploading two years ago gets a two-year-old aggregate presented as current, with nothing in the column marking it as stale.

Once three models read the same stored vector, the encoder is an interface and retraining it is a breaking change for consumers who did not ask for it. The terms are versioned namespaces, a pin per consumer, and a stated deprecation window.