An etcd member backing a Kubernetes cluster starts rejecting writes with "mvcc: database space exceeded". What causes this, and what is the difference between compaction and defragmentation in fixing it?

## Why the database grows etcd never updates a key in place. Every mutation appends a new version tagged with a globally increasing revision, and old versions stay readable so clients can watch or read from a past revision. That gives Kubernetes its `resourceVersion` semantics and resumable watches — but it means a workload that rewrites the same object repeatedly grows the store even though the key count is constant. Typical growth drivers: - **Events** — high-churn, short-lived objects; large clusters give Events their own etcd instance via `--etcd-servers-overrides`. - **Hot-looping controllers or operators** patching status every second. - **Lease renewals** from node heartbeats and leader elections. - **Large or numerous custom resources**, or huge ConfigMaps and Secrets (etcd is tuned for small values; the per-value limit is 1.5 MiB and even approaching it is a smell). ## The quota and the alarm etcd enforces `--quota-backend-bytes`, default 2 GiB (values above 8 GiB are discouraged — bigger databases mean slower startup, defrag and restore). On exceeding it, etcd raises a cluster-wide **NOSPACE alarm** and refuses all writes until it is disarmed. This is a safety valve, not a crash: `kubectl get` still succeeds while `kubectl apply` fails with `etcdserver: mvcc: database space exceeded`. Symptoms look like a frozen control plane — no scaling, no rollouts, no new Pods, Leases failing to renew. ## Compaction: dropping history Compaction removes all versions older than a given revision, keeping the latest version of each live key. `etcdctl compact 12345` does it explicitly; more usefully it happens automatically: - **kube-apiserver** compacts on a timer, `--etcd-compaction-interval`, default 5 minutes. In a normal cluster this is what actually keeps history bounded. - **etcd itself** can compact with `--auto-compaction-mode=periodic --auto-compaction-retention=1h` (or `revision` mode). Both compacting is harmless — the later revision simply wins. The visible cost: a watcher resuming from a compacted revision gets `required revision has been compacted`. Kubernetes clients recover by doing a full relist and restarting the watch — the "watch expired, relisting" behaviour informers show. Over-aggressive retention therefore appears as expensive relist storms against the API server. Crucially, **compaction does not shrink the file**. Freed pages are marked reusable inside etcd's backing database, so on-disk size stays at its high-water mark and future writes reuse the space. If you are already at quota, compaction alone may not get you back under it. ## Defragmentation: shrinking the file `etcdctl defrag --endpoints=<one member>` rewrites the backend file, packing live pages together and releasing the tail to the filesystem. Points to state in an interview: - It is a **per-member, local** operation — run it against each endpoint, not once for the cluster. - It is **blocking**: that member stops serving while it rewrites, which for a multi-GB database is seconds to a minute. One member at a time so quorum survives, and do the **leader last** (or move leadership first) to avoid an election mid-defrag. - It is only worth doing when there is a real gap between `dbSize` and `dbSizeInUse` (both shown by `etcdctl endpoint status -w table`). Nightly defrag of a healthy small database is unnecessary churn. ## The recovery runbook 1. Confirm: `etcdctl alarm list` shows NOSPACE; `endpoint status` shows dbSize near quota and dbSizeInUse much smaller. 2. Compact to the current revision (reported by `endpoint status`), or wait for the API server's automatic compaction. 3. Defrag each member in turn, leader last. 4. `etcdctl alarm disarm` — writes do not resume until you do, and this is the step people forget. 5. Root-cause the growth: check Event volume, look for controllers hot-looping on status updates, consider `--etcd-servers-overrides` to split Events onto their own etcd, tune `--event-ttl`, and re-check auto-compaction retention. Raising `--quota-backend-bytes` is a stopgap: it buys time, but a growing database also slows snapshots, restores and startup. ## Preventive posture Alert on `etcd_mvcc_db_total_size_in_bytes` approaching the quota and on the gap between total and in-use size. Watch disk fsync latency (`etcd_disk_wal_fsync_duration_seconds`) too — slow disks and a bloated database produce the same user-visible symptom of a sluggish control plane.