A live tier's key convention must gain an owning prefix while callers keep reading and writing — how do you run that change, and how do you know the old shape is dead?

## Why a convention is hard to change once it is carrying traffic The key is the only address the store understands. Change the string and you are not renaming a column — you are pointing at a different entry, one that does not exist yet. Meanwhile callers are reading and writing continuously, several versions of the application are live at once during any rolling deployment, and the store will happily let every shape coexist because it enforces nothing. There is no migration tool here for the same reason there is no schema. ## Step zero: get the key into one place Before anything else, every key must be produced by a single builder — owner, entity, identifier in, string out. If keys are concatenated inline at forty call sites, the migration is an archaeology exercise and you will miss some. This step is often most of the work, and it is worth doing even if the convention never changes again. ## The order of deployment 1. **Readers that understand both.** Deploy a release that reads the new key and, on a miss, reads the old one. Writers are untouched, so this release is inert in production: every read misses the new shape and falls back. Let it bake. 2. **Writers to the new shape.** Flip writing. New entries now appear under the new key only; existing entries still answer through the fallback. 3. **Deal with what is left.** Either let entries age out, or run a deliberate pass that reads an old entry and writes it under the new key. 4. **Remove the fallback**, then remove the old entries, then remove the builder's old branch — in that order, so that nothing is deleted while any reader can still want it. Doing this in the other order — writers first — means a reader from the previous release looks for an old key that nobody writes any more, and reports a miss for data that exists. ## The window in which both shapes exist This is the part candidates skip, and it is where the incident comes from. Between steps two and four, one logical thing has two possible addresses, and they can disagree: - an update written only in the new shape leaves a stale value under the old key, which an older reader still trusts; - an update written in **both** shapes costs two writes with no atomicity between them, so a failure between them leaves the shapes disagreeing; - a deletion is worse than an update: delete only the new key and the old one resurrects the value through the fallback. Pick one discipline and state it: either **write both and delete both**, accepting the failure window between the two operations, or **write only the new shape and treat old entries as read-only leftovers**, which is only safe once no caller can still write the old shape. Deleting must follow whichever rule you chose. ## How you know the old shape is dead Instrument the fallback. Increment a counter every time a read misses the new key and finds an old one, and label it by the calling site if you can. Then: - watch the counter fall as traffic naturally rewrites entries; - when it reaches zero, keep waiting — an entry read once a quarter has not been observed yet; - for a keyspace whose entries carry a lifetime, the honest waiting period is **longer than the longest lifetime in play** after the last non-zero reading; - for entries with no lifetime, waiting proves nothing, and you need the deliberate pass instead. What you should not do is confirm it by reading the code. A code search tells you nothing about what is in the store, and this class of store never had a schema to tell you either. ## What varies between stores, and why it matters to the plan - **Moving an entry server-side.** Some stores offer a single server-side step that gives an entry a new name; others do not, and the move becomes read-then-write in the client. Do not build the plan on the step existing. - **Where the keyspace is split across nodes**, the old and new key may sit on different nodes, so any step that wants to touch both at once may not be expressible. Which node a key lands on is a separate subject; the consequence for you is simply that a migration step should touch one key at a time. - **Finding the leftovers** is its own problem. Enumerating what remains under the old shape on a serving store is a traversal question, and it is the reason attrition through lifetimes is so attractive when it is available. ## The senior signal A candidate who has only read about this describes the new convention. One who has run it describes readers before writers, the window where both shapes exist, the rule for deletes inside that window, and the counter that gave them permission to delete the old branch.