Scripted Data Changes

## What a data migration is A Django **migration** is a Python file in an app's `migrations/` package holding a `Migration` class with two lists: `dependencies` (which migrations must run first) and `operations` (what to do). Most operations are **schema** operations such as `AddField` or `RemoveField`, and `makemigrations` writes them for you by comparing your models with the state recorded in earlier migrations. A **data migration** changes rows rather than structure: filling a new column, merging duplicates, normalising values. `makemigrations` never generates one, because a change to rows leaves the models identical and there is nothing to detect. You write it by hand, and the main tool is the **`RunPython`** operation (with `RunSQL` as the raw-SQL alternative). ## Step by step for the full_name split Suppose a `Customer` model in an app labelled `crm` has a `full_name` field and you want separate `first_name` and `last_name` fields. 1. Add `first_name` and `last_name` to the model and run `makemigrations`; Django writes a schema migration (say `0007`) with two `AddField` operations. 2. Run `python manage.py makemigrations crm --empty --name split_full_name`. Django creates `0008_split_full_name.py` with `dependencies` pointing at `0007` and an empty `operations` list. 3. Write a **forwards function** above the `Migration` class. Django calls it with two arguments: `apps`, a registry of **historical models** as they exist at this point in the migration history, and `schema_editor`, the object that issues DDL and exposes the current connection. 4. Inside it, fetch the model with `apps.get_model("crm", "Customer")`, loop over the rows, split the name, and write the new values back, preferably in batches. 5. Write a **reverse function** that rebuilds `full_name` from the parts, so `migrate crm 0007` can undo the step. 6. Add `migrations.RunPython(forwards, reverse)` to `operations` and run `python manage.py migrate`. 7. Later, once nothing reads `full_name`, remove the field in a third migration. ## How the three migrations line up | Migration | Operation | What it does | |---|---|---| | `0007` | `AddField` x2 | adds the empty `first_name` and `last_name` columns | | `0008` | `RunPython` | copies data out of `full_name` into the new columns | | `0009` | `RemoveField` | drops `full_name` once the application no longer uses it | Keeping the data step in its own migration is the pattern Django's documentation recommends: data migrations are "best written as separate migrations, sitting alongside your schema migrations". It keeps each file reviewable and lets you roll back the data step independently. ## What goes inside the function - Use **`apps.get_model()`**, never `from crm.models import Customer`; the imported class reflects today's code, not the schema at migration `0008`. - Route queries through **`schema_editor.connection.alias`** with `.using(alias)` so the migration writes to the database that `migrate` is targeting, not always `default`. - Prefer **`bulk_update(objs, fields, batch_size=...)`** or a queryset `update()` over calling `save()` per row; historical models do not have your custom `save()` anyway. - Stream large tables with **`.iterator(chunk_size=...)`** rather than loading every row into memory at once. - Keep the function a **module-level function** in the migration file, not a lambda and not a method imported from application code that may change or disappear. ## Common beginner mistakes - Expecting `makemigrations` to notice that data needs changing. - Importing the model from `models.py`, which works today and breaks a fresh install months later. - Omitting the reverse function, which makes the migration impossible to unapply. - Putting the `AddField`, the data copy and the `RemoveField` in one file, so a failure or rollback cannot separate them. - Calling application services or signals from the migration, coupling a frozen historical step to code that will keep evolving. Once written, the data migration is an ordinary node in the migration graph: `showmigrations` lists it, `migrate` applies it exactly once per database, and the `django_migrations` table records that it ran.