Your CustomResourceDefinition currently offers version v1alpha1 and you want to introduce v1beta1 without breaking existing objects or existing clients. Explain what the `served` and `storage` flags mean on each version entry, and when you need a conversion webhook.
answer
- served = clients may use it; storage = the one form in etcd
- exactly one storage: true
- flipping storage does not rewrite existing objects
- migrate objects, then prune status.storedVersions
- strategy None only for identical schemas; Webhook must be lossless both ways
basics
~20 sserved: true means clients may read and write that version; exactly one version has storage: true and is the form written to etcd. Objects are converted between versions on the fly — trivially if the schemas are compatible (strategy: None), otherwise you must run a conversion webhook.
solid answer
~50 sA CRD lists versions independently. **`served`** controls whether the API server exposes `/apis/<group>/<version>/…` for that version. **`storage`** marks the single version whose representation is written to etcd — exactly one may be true. So you add `v1beta1` with `served: true, storage: false` first. Clients can now use either version; every object is still stored as v1alpha1 and converted on read. When you are ready, flip `storage` to v1beta1: newly written objects are stored as v1beta1, while **already-stored objects stay in the old form** until something rewrites them. Conversion is declared in `spec.conversion`. **`strategy: None`** just relabels apiVersion and is only correct when the schemas are structurally identical. Any real change — renamed or restructured fields, split or merged values — requires **`strategy: Webhook`**, a TLS endpoint that converts objects both ways. Before you can drop a version, all stored objects must be migrated and `status.storedVersions` must no longer list it.
code
yaml · 41 linesapiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.com
spec:
group: example.com
scope: Namespaced
names: { plural: databases, singular: database, kind: Database }
conversion:
strategy: Webhook
webhook:
conversionReviewVersions: ["v1"]
clientConfig:
service:
name: db-conversion
namespace: db-system
path: /convert
caBundle: LS0tLS1CRUdJTi...
versions:
- name: v1alpha1
served: true
storage: false
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
size: { type: string }
- name: v1beta1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
storageSize: { type: string }go deeper
Know that served means the version is exposed to clients and that exactly one version is the storage version written to etcd.
Walk the add-serve, migrate-clients, flip-storage sequence and say when strategy None is safe versus when a webhook is required.
Add storage migration, status.storedVersions pruning before version removal, round-trip losslessness with hub-and-spoke conversion, and the availability impact of the conversion webhook on reads.
Set the API evolution policy: what changes are allowed within a version, when a new version is justified given the conversion-webhook operational cost, and how deprecation windows are communicated and enforced across teams.
## One object, several representations A CRD's `spec.versions` is a list, and each entry carries its own schema, its own `additionalPrinterColumns` and its own `subresources`. The key insight is that there is only ever **one object** in etcd; the versions are alternative representations of it, and the API server converts between them on every read and write. Two flags control the lifecycle: - **`served: true|false`** — whether the API server exposes that version's endpoint. A version with `served: false` still exists in the CRD (and may still be the historic storage version) but clients cannot use it. Turning `served` off is how you retire a version without deleting data. - **`storage: true|false`** — exactly one version must be `true`. When an object is written, the API server converts it to this version and stores that form. Making a version the storage version is not retroactive. ## Adding a version safely The standard sequence for introducing `v1beta1` alongside `v1alpha1`: 1. **Add** the `v1beta1` entry with `served: true, storage: false`, plus a conversion strategy if the schemas differ. Both endpoints now work; every object is still stored as v1alpha1 and converted up on read. 2. **Migrate clients** — controllers, GitOps repos, CI — to v1beta1. Nothing breaks, because both are served. 3. **Flip storage**: `v1beta1` gets `storage: true`, `v1alpha1` gets `storage: false`. From now on, writes store v1beta1. Existing objects **remain stored as v1alpha1**; the API server converts them on read. 4. **Storage migration**: force every object to be rewritten so it is persisted in the new version. The simple way is to read and write each object (`kubectl get <res> -A -o json | kubectl replace -f -`); the supported tool is the storage version migrator, driven by `StorageVersionMigration` resources in newer clusters. 5. **Prune `status.storedVersions`**: the API server tracks every version any object has been stored in. Until v1alpha1 is removed from that list — which you may only do after migration — you cannot remove the version, because the API server would be unable to decode objects still in that form. 6. **Stop serving** v1alpha1 (`served: false`), then remove the entry entirely. Skipping step 4 or 5 is the classic failure: you delete the old version, and objects still stored in it become undecodable. ## Conversion strategies `spec.conversion.strategy` takes two values. **`None`** performs no transformation at all — the API server simply relabels `apiVersion`. It is correct **only** when the versions are structurally identical, or differ merely by additive optional fields where the missing data is acceptable. Using `None` across a renamed field silently loses data: writing v1beta1 with `spec.storageSize` and reading v1alpha1 expecting `spec.size` yields an object with the field pruned. **`Webhook`** points at a TLS endpoint (a Service reference plus `caBundle`, exactly like admission webhooks) that receives a `ConversionReview` containing a list of objects and the `desiredAPIVersion`, and returns the converted objects. Requirements: - It must convert **in both directions**, since reads may request any served version. - It must be **lossless in practice**: the usual technique is a *hub-and-spoke* model where every version converts to and from one internal hub version, and any field that exists only in a newer version is round-tripped through an annotation so a v1beta1 → v1alpha1 → v1beta1 trip does not destroy data. - It is on the read path for *every* request to a non-storage version, including list operations that may pass many objects at once, so it must be fast and highly available. If it is down, objects in the non-storage version cannot be read at all — including by your own controller. Because `apiextensions.k8s.io/v1` requires the schema per version, the API server validates each representation against its own schema after conversion. ## Compatibility discipline Since the conversion webhook is real operational risk, the strongest move is to avoid needing one: prefer **additive, optional** changes within a version, and use a new version only for genuinely breaking restructurings. Kubernetes' own API convention — alpha may break, beta must be convertible, GA is stable — is a good rule to adopt for your CRDs too. Note also that `served: false` on an alpha version is a legitimate way to force migration in a pre-production platform. One more operational detail: the CRD's `status.conditions` will report `NonStructuralSchema` or conversion problems, and `kubectl get crd <name> -o jsonpath='{.status.storedVersions}'` is the single most useful command when planning a version removal.
- You flip storage from v1alpha1 to v1beta1. What happens to the objects already in etcd?Nothing immediately — they remain stored in the v1alpha1 representation and are converted on every read. Only a write rewrites an object at the new storage version. Until you force a rewrite of every object and prune status.storedVersions, removing v1alpha1 from the CRD would leave those objects undecodable.
- Why must a conversion webhook be able to round-trip an object without losing data?Because a client may read an object in an older version and write it back, or a controller may operate on the older version entirely. If the older schema has no home for a newer field, that field is lost on the round trip. The standard fix is a hub-and-spoke conversion where unrepresentable fields are stashed in an annotation and restored on the way back.
- When is conversion strategy None acceptable?Only when the versions are structurally the same, or differ by optional additive fields whose absence is harmless in the older representation. Any rename, restructuring, type change or semantic change requires a webhook, because None does nothing but relabel apiVersion and the mismatched fields are pruned by the target version's schema.
Think of a document archive that accepts submissions in several languages but files everything in one: readers can request any translation, older files stay in the language they were filed in, and you cannot retire a language until every file in it has been re-filed.
saying these in an interview costs you the question
- Thinking more than one version can have storage: true
- Assuming flipping the storage version rewrites existing objects
- Removing an old version without migrating stored objects and pruning status.storedVersions
- Using strategy None across renamed or restructured fields and expecting data to survive
- Forgetting that a conversion webhook sits on the read path, so its downtime makes objects unreadable