Why does the Kubernetes API server refuse to remove v1alpha1 from a CRD's spec.versions, and how do you migrate stored objects so the removal succeeds?

## What `status.storedVersions` records Every CustomResourceDefinition (CRD) has exactly one **storage version** — the `spec.versions` entry with `storage: true` — and the API server encodes each write in it. Changing the storage version does not re-encode the objects already in etcd, so over a CRD's life etcd can hold objects in several versions at once. The CRD tracks that in **`status.storedVersions`**: a list of every version that has ever been the storage version. The apiextensions API server **only adds** to it — whenever a CRD is created or updated and its storage version is not yet listed, that version is appended. Nothing in the API server checks etcd to remove an entry; removing one is the job of whoever migrated the data. For the search-autocomplete team's `suggestindexes.autocomplete.example.com`, a CRD that started at `v1alpha1` and now stores `v1` shows `storedVersions: ["v1alpha1", "v1"]`. ## Why removal is refused CRD validation enforces two rules on that list: - It must include the current storage version ("must have the storage version v1"). - Every listed version must still appear in `spec.versions`. Dropping `v1alpha1` fails with: `v1alpha1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1alpha1 and removes v1alpha1 from status.storedVersions`. The reason is decoding. The converter accepts only source versions that are listed in `spec.versions`. If an object stored as `v1alpha1` is read after that entry is gone, the request fails with "request to convert CR from an invalid group/version", and a list that contains even one such object fails as a whole — every controller that lists the resource stalls. ## Migrating with `StorageVersionMigration` In v1.37 the **StorageVersionMigrator** feature is GA and kube-controller-manager runs the `storage-version-migrator-controller`. You ask it to migrate a resource with an object in `storagemigration.k8s.io/v1`: ```yaml apiVersion: storagemigration.k8s.io/v1 kind: StorageVersionMigration metadata: name: suggestindexes-to-v1 spec: resource: group: autocomplete.example.com resource: suggestindexes ``` What happens next: 1. The controller sets a `StorageMigrating` condition on the CRD, recording the CRD's current `metadata.generation`. 2. It sends a **no-op patch** to every object of the resource. The API server re-encodes each one in the storage version, and because the new bytes differ from the old ones, the write reaches etcd. 3. When all patches succeed, the `StorageVersionMigration` gets a `Succeeded` condition and the controller sets `storedVersions` to just the storage version — but only if the CRD's generation still matches the one it recorded. If someone changed the CRD spec mid-run, it reports that the stored versions were not updated "due to a generation mismatch", and you run a new migration. ## Migrating by hand On a cluster without that controller, do the same thing yourself: read every object and write it back unchanged, then patch the CRD status. - `kubectl get <plural>.<group> -A -o json | kubectl replace -f -` rewrites every object; a `409 Conflict` on a busy object only means it was written meanwhile, which already re-encoded it. - `kubectl patch customresourcedefinition <name> --subresource=status --type=merge -p '{"status":{"storedVersions":["v1"]}}'` records the result. A JSON merge patch replaces the whole list. - Merely reading objects migrates nothing — reads convert in memory only. ## The safe retirement sequence 1. Ship `v1` with `storage: true` and, if schemas differ, a working conversion webhook. 2. Mark `v1alpha1` `deprecated: true` so clients see warnings. 3. Set `served: false` for at least one release so stray clients fail loudly while the data is still readable. 4. Migrate stored objects and confirm `storedVersions` is `["v1"]`. 5. Remove the `v1alpha1` entry, and its code path in the conversion webhook. ## Pitfalls | Mistake | Consequence | |---|---| | Trimming `storedVersions` without migrating | Validation lets you delete the version; old objects become unreadable and lists fail | | Migrating while the conversion webhook is down | Every patch needs conversion, so the migration fails | | Editing the CRD spec during migration | The migrator refuses to update `storedVersions` | | Removing the version in the same release that stops serving it | No window to catch clients still using it | If an old version was removed too early, re-adding its entry to `spec.versions` (with `served: false` and `storage: false`, plus its conversion code if you use a webhook) makes the stored objects decodable again, after which you can migrate properly.