Why can cached clause embeddings survive a retrain that invalidates every cached label?

## Two places to cache on one path A clause-scoring request runs through stages: segment the document, encode each span into a vector, apply a classifier head to that vector, post-process into labels and risk flags. A cache can sit at the end of that path or in the middle of it, and the two choices behave differently under change. | What is stored | Key pins | Invalidated by | Entry size | Reuse | |---|---|---|---|---| | finished labels | span hash + scoring-function version | any change to encoder, head, label set or preprocessing | tens of bytes | one consumer | | the span's vector | span hash + encoder version | a change to the encoder only | kilobytes | every head reading that encoder | The difference in the second column is the whole answer. A label is the output of everything; a vector is the output of one stage. The narrower the dependency a key pins, the more changes the entry survives. ## Why the encoder is usually the stable half In a document-review workflow, the things that change often are the ones close to the business: a new clause category is added to the taxonomy, a head is refitted on the last quarter's reviewer verdicts, a risk threshold moves. The text encoder is expensive to train, is shared by several heads, and is replaced rarely. So the common retrain bumps the scoring-function version, empties the prediction namespace and leaves the embedding namespace exactly as it was. That is a condition, not a law. Where the encoder is fine-tuned on the same cadence as the head, the embedding cache is invalidated just as often and buys nothing but memory pressure. Check which half of your stack actually moves before designing around this. ## What an embedding hit actually buys An embedding hit does not remove the scorer from the request path; it shortens it. A request whose label is a miss but whose vector is a hit skips the encoder pass and still runs the head, which is the cheap end of the computation. So the payoff is proportional to how much of the forward pass the encoder represents - substantial for a deep text encoder over a long clause, negligible where the model is small. The second payoff is across consumers. One namespace of span vectors can serve a contract-type head, a clause-category head and a risk head at once, so an expensive encode is paid for once and reused three times. The price is that the encoder version becomes a contract between those consumers: whoever replaces it invalidates everyone's entries simultaneously. ## The arithmetic of storing vectors Size is what usually decides this. A vector of 768 dimensions stored as 32-bit floats is about 3 KB. Holding one for each of 600,000 distinct spans is roughly 1.84 GB, against perhaps 60 MB for the same spans' labels. Two consequences follow: 1. **The same memory holds far fewer embeddings than predictions**, so the embedding cache is normally scoped to the head of the key distribution while the prediction cache can cover more of the tail. 2. **Fetching an entry is no longer free.** Moving 3 KB across the network costs real time, and for a short span on an idle accelerator, re-encoding can genuinely be cheaper than the lookup. Measure the two before assuming the cache wins. Storing the vectors at reduced precision halves the footprint and changes the stored value, so it has to be treated as a change to what is cached rather than a storage detail. ## A different question from putting embeddings in the feature store Whether an embedding belongs in the online store as a model *input* - how it is refreshed, who owns its definition, what a stale vector means as a feature - is a separate subject. This is narrower: a memo of a deterministic function, so that the same span is not encoded twice. If the vector is a feature with a life of its own, the version-and-TTL reasoning above is not the mechanism governing it. ## When not to bother Skip the embedding cache when the encoder is retrained as often as the head, when the encode is cheap relative to a multi-kilobyte fetch, or when the key space is so flat that few spans are ever encoded twice. Skip it also when only one head consumes it and the finished-label cache already covers that traffic - in that case you are paying vector-sized memory for a benefit the smaller cache is already delivering.