Operators and CRDs

A CustomResourceDefinition registers a new resource type with the Kubernetes API server. You immediately get REST endpoints for it, storage in etcd, kubectl support, watches, RBAC and labels — but nothing acts on the objects until you run a controller for them.

An Operator is a custom Kubernetes resource type plus a controller running in the cluster. Users declare the desired database in a custom object; the controller continuously drives the real system to match that spec, automating install, backup, upgrade and failover.

The API server authenticates and authorizes, then runs mutating admission (webhooks may patch the object), then schema validation, then validating admission (accept or reject only), then writes to etcd. Mutation runs first so validators judge the final stored object.

The schema validates types, required fields and constraints at admission, applies declared defaults, and prunes — silently strips — any field it does not describe. To keep arbitrary content, mark that subtree with x-kubernetes-preserve-unknown-fields: true.

It splits the object into two write paths: writes to the main endpoint ignore status, and writes to /status change only status. It also makes metadata.generation increment only on spec changes, and lets RBAC grant status updates separately.

A CRD stores exactly one version, but old objects stay in etcd as last written; the API server converts them in memory whenever a request needs another version. Strategy None only rewrites apiVersion, so it suits identical schemas only.

Level-triggered means the controller reads current desired and actual state on every pass and closes the gap, instead of reacting to individual change events. Reconcile may run many times for one change, in any order, after restarts, so it must be safe to repeat and must not depend on having seen previous events.

Set an ownerReference on each created object pointing at the custom resource. The garbage collector then deletes dependents automatically when the owner is deleted. Owner and dependent must be in the same namespace, and a cluster-scoped object cannot be owned by a namespaced one.

The API server sets metadata.generation and increments it when desired state changes. The controller copies the generation it acted on into status.observedGeneration, so readers know the status is current only when the two numbers match.

Fail means a webhook error, timeout or unreachable backend causes the API request to be rejected; Ignore means the API server logs it and admits the request as if the webhook had approved it. Fail is safe for correctness, Ignore is safe for availability.

Build an operator when the application needs ongoing, application-specific decisions — ordered upgrades, failover, backup and restore, resharding — that no template can make. If install-and-forget is enough, a chart plus StatefulSet is cheaper. The operator costs you a privileged, always-running component to write, secure, upgrade and monitor.

A finalizer is a string in metadata.finalizers that blocks final deletion. A delete request only sets deletionTimestamp; the object persists until every finalizer is removed. The controller does its cleanup, then removes its own finalizer. Stuck Terminating usually means the responsible controller is gone or failing.

A shared informer watches the API server, keeps an in-memory cache of objects, and on each event pushes only the object's key onto a rate-limited work queue. Workers pop keys and reconcile, reading from the cache. The queue deduplicates keys, bounds concurrency, and provides retry with backoff.

Conditions are status entries a custom resource's controller writes, each with a type such as Ready, a status of True, False or Unknown, a reason and a message. Read them with kubectl describe or jsonpath; block on them with kubectl wait --for=condition=.

sideEffects declares whether the webhook changes state outside the object under review. In admissionregistration.k8s.io/v1 it must be None or NoneOnDryRun. For a server dry-run request, the API server skips None-declared-unsafe webhooks and calls the others with dryRun: true set in the AdmissionReview.

The API server only calls webhooks over HTTPS, and it verifies the server certificate against the caBundle in the webhook configuration, requiring a SAN matching <service>.<namespace>.svc. On expiry the handshake fails, so requests are rejected under failurePolicy: Fail or silently unchecked under Ignore.

mgr.GetClient() serves Get and List from informer caches but sends writes to the API server, so a read can lag your own write, and the first read of a new kind starts a cluster-wide informer. mgr.GetAPIReader() reads live and caches nothing.

Namespaced objects live in a namespace, are named uniquely per namespace, and are covered by namespaced Roles and namespace deletion. Cluster objects are global, unique cluster-wide, and require ClusterRoles. Choose namespaced for tenant-owned resources, cluster for shared cluster-level configuration.

The levels run: 1 basic install, 2 seamless upgrades of the managed application, 3 full lifecycle (backup, restore, failover), 4 deep insights (metrics, alerts, logs), 5 autopilot (auto-scaling, auto-tuning, auto-remediation). Use them to check the operator actually covers the operations you would otherwise do by hand.

Returning an error requeues the key through a rate limiter with exponential backoff, so retries slow from milliseconds to minutes. Returning RequeueAfter schedules a re-check at a fixed delay for polling. Returning an empty result with no error drops the item and resets its backoff. Never retry in a loop inside reconcile.

Owns cannot help because the shared ConfigMap has no single controlling owner. Add Watches(&corev1.ConfigMap{}, handler.EnqueueRequestsFromMapFunc(fn)), where fn uses a field index on spec.profileRef to list referencing GameSessions and return their requests. Scope the ConfigMap cache with a selector.

served: true means clients may read and write that version; exactly one version has storage: true and is the form written to etcd. Objects are converted between versions on the fly — trivially if the schemas are compatible (strategy: None), otherwise you must run a conversion webhook.

The CRD's status.storedVersions still lists v1alpha1, so etcd may hold objects in it. Re-encode every object in the storage version with a StorageVersionMigration or no-op writes, trim storedVersions to v1, then delete the version entry.

Every request needing conversion fails or slows: lists, watches, controllers' informers, writes, namespace deletion and migrations. There is no failurePolicy to skip it, so run the webhook redundantly with a PodDisruptionBudget, pure fast code and rotated TLS.

Every matching write now waits on a network call, so webhook latency becomes API latency and webhook downtime can block Pod creation. Scope with rules and selectors, exclude kube-system and the webhook's own namespace, keep timeouts at a few seconds, run it HA, and ship with failurePolicy Ignore before flipping to Fail.

Add additionalPrinterColumns to the CRD version: each entry has a name, a type, and a jsonPath into the object. The API server renders the table server-side, so every client that asks for a table sees the same columns.

Setting deprecated: true keeps the CRD version fully working but adds a warning to every response at that version, optionally replaced by deprecationWarning. Setting served: false removes the version's endpoints, so requests get 404 while stored objects stay readable.

envtest starts real etcd and kube-apiserver binaries locally, installs your CRDs and returns a rest.Config for your manager. It runs no kube-controller-manager, scheduler or kubelet, so there is no garbage collection, no Pods running and no finished namespace deletion.

The CRD declares subresources.scale with specReplicasPath, statusReplicasPath and, for an HPA, labelSelectorPath. The controller applies the desired replicas and writes the actual replica count and a string label selector for the Pods it owns.