A release ships application code together with a database migration that renames a column, and the release turns out to be bad. Why can you no longer simply redeploy the previous version, and how should that migration have been sequenced so the code stays rollback-safe?

## Why schema changes are what actually blocks rollback Most rollbacks are trivial: point production at the previous artifact. Schema and data are what turn that into an outage, because the database is shared mutable state that both versions of the code must be able to use. A code deploy is reversible; a `DROP COLUMN` is not. When a release runs `ALTER TABLE ... RENAME COLUMN` and ships code that uses the new name, the two are welded together. Rolling back the code alone leaves old code against a new schema - every statement referencing the old name fails. There is now no version of your code that works with the database as it currently stands except the bad one. ## Why the "down migration" is not the answer Migration tools let you write a reverse step, and that creates a false sense of safety: - **It cannot restore data.** Reversing a `DROP COLUMN` re-creates an empty column. Everything that was in it is gone unless you separately preserved it. - **It loses the writes since.** Rows created or updated while the new version was live wrote to the new shape. Reversing the schema abandons them or leaves them inconsistent. - **The DDL itself is a risk.** On a large table, adding or dropping a column, adding a `NOT NULL` constraint, or rebuilding an index can lock or rewrite the table for a long time. Running that under incident pressure adds a second, larger outage on top of the first. - **It takes as long as it takes.** A rollback whose duration is proportional to your table size is not a rollback. So the reverse migration is a development convenience. In production, treat the schema as append-only and forward-moving, and make the *code* the reversible part. ## Expand-contract, step by step The rule: **every schema state must be usable by both the version before and the version after it.** For a column rename from `email` to `email_address`: 1. **Expand.** Add `email_address` as nullable with no constraint. Non-destructive; the running code does not know it exists. Deploy the migration on its own. 2. **Dual-write.** Deploy code that writes both columns and still reads `email`. Rolling back to step 1's code is safe - it only ignores the new column. 3. **Backfill.** Copy existing rows in bounded batches, throttled so it does not saturate the database. Idempotent and resumable, so a failure mid-run is not a crisis. Verify with a count of rows still unmigrated. 4. **Switch reads.** Deploy code that reads `email_address` and still writes both. Rolling back to step 2's code is safe, because both columns are still current. 5. **Stop writing the old column.** Now the old column stops receiving writes - this is the first point where rolling back past it means losing data written since. 6. **Contract.** Drop `email`, after you are confident you will not roll back past step 5, and after any consumers you do not control (reporting jobs, exports, another service reading the same table) have moved. Steps 1-4 are all reversible by redeploying the previous artifact. That is the whole point: the reversible thing is the code, and the schema only ever moves in a direction that both versions tolerate. ```sql -- 1. expand: additive, no rewrite of existing rows in modern engines ALTER TABLE users ADD COLUMN email_address text; -- 3. backfill in bounded batches, resumable UPDATE users SET email_address = email WHERE email_address IS NULL AND id >= 1 AND id < 10000; -- 6. contract: only after the rollback window has closed ALTER TABLE users DROP COLUMN email; ``` ## The rules worth stating in an interview - **Never ship a destructive DDL in the same deploy as the code that depends on it.** Migration and code are separate, ordered deploys. - **Every migration must be compatible with the previously deployed version of the code (N-1).** Expansion leads the code; contraction trails it. - **Adding a `NOT NULL` column with no default is the classic trap** - the old code does not know the column and its inserts start failing. Add nullable or with a default, backfill, then constrain. - **The rollback window sets the contract schedule.** If you are willing to roll back for a week, the drop happens a week later, not in the same change. ## The honest cost A one-line rename becomes four to six deploys over several days, plus a backfill to babysit and dual-write code to clean up afterwards. Teams skip it because it feels like ceremony for a trivial change. The trade is explicit: you are paying a few days of process to keep every release in the sequence rollback-safe. On a service where a bad release means a five-minute rollback instead of an hour of forward debugging, that trade is nearly always worth it - and the discipline only helps if it is the default, because you cannot retrofit it during the incident.