You roll a service back to its previous version an hour after a bad release. During that hour the new version wrote records containing a field the older version knows nothing about. What problems does the rollback now create, and what limits how far back you can safely go at all?

## The rollback moves the code, not the data A rollback restores a binary. It does not restore the database rows, the topic offsets, the cache entries or the third-party side effects the new version produced while it was live. Those are now inputs to the old version, and the old version was written before they existed. The interesting failures are the quiet ones. ## Four ways forward-written data hurts **1. Hard read failures.** A strict deserializer that rejects unknown fields will throw on every record the new version wrote. Errors cluster on recently touched entities, which makes the symptom look random until you notice they all share a write timestamp inside the incident window. **2. Silent field loss.** This is the dangerous one. The old code reads a record, ignores the field it does not know, and writes the whole object back. The new field is gone - no error, no alert, and no way to tell later which rows lost it. If the new field carried anything customers can see, you now have a data-integrity incident hiding behind a resolved availability incident. **3. Unknown enumerated values.** The new version introduced a status, type or state that the old code has no branch for. Depending on the language it either throws or falls into a default that means something different, quietly mis-classifying those records. **4. Residue that keeps failing after the rollback.** Caches hold serialized objects in the new layout; queues hold messages in the new schema; a search index holds documents with a new mapping. The rolled-back code keeps hitting them and keeps failing, so the SLI does not recover and it looks like the rollback did not work. The fixes are unglamorous: version cache keys so a release change invalidates them implicitly, park un-parseable messages in a dead-letter destination rather than blocking the consumer, and be prepared to drain or reprocess. ## Designing so the old version survives the new one's leftovers The property you want is **compatibility in both directions**, and it is engineered before the release, not after: - **Tolerant reader.** Both versions ignore fields they do not recognise instead of failing on them. This is what makes additive change safe. - **Preserve unknown fields on write.** If your serialization can round-trip unrecognised fields, the silent-loss problem disappears. If it cannot, avoid read-modify-write over whole objects or accept that rollback loses those fields. - **Additive-only message and event schemas** during the window in which you might roll back, with a registry check if you have one. - **Version keys, not just values.** Cache keys and index names that include a schema or release version make stale-format entries unreachable rather than poisonous. - **Never make a new field required immediately.** Optional first, then required after the rollback window. ## What your rollback horizon actually is Teams say "we can always roll back". Almost nobody can. The horizon is the oldest version that can still run correctly against *today's* state, and it is shortened by: - **Completed contract steps** - once a column, field or endpoint the older code used is gone, that code cannot run. - **New required fields or constraints** the older version does not populate. - **Message and API contracts** - a downstream dependency that has moved on, or a partner API version the old client no longer speaks. - **Rotated or expired credentials and pinned certificates** baked into the older artifact. - **Artifact availability** - a release whose build no longer exists is not a rollback target. For most services the honest horizon is one release back, sometimes a small number of releases, and it changes every time a contract step lands. The practice worth naming in an interview is: state the horizon explicitly, keep it in the release notes or the service's operational metadata, and verify it - the cheapest verification is that your pre-production environment routinely runs the previous version against the current schema. ## Cleaning up afterwards A rollback that lost or mangled data is not finished when the SLI recovers. The follow-up work is part of the incident: identify the affected write window, quantify the rows or messages involved, decide whether to repair them from another source (an event log, an audit table, a backup) or accept the loss, and record it. Teams that skip this step discover it months later as a support ticket nobody can explain.