skip to content

etcd Backup & Recovery

7 roadmaps4 questions

What you actually restore when a cluster is gone: object state from an etcd snapshot, application objects and volume data through Velero and CSI snapshots, or manifests re-applied from Git. Interviewers ask to find who has rehearsed a recovery.

part ofKubernetesoverview, primer and where to startread it →
on this pageshow

explore

questions

4

When a Kubernetes cluster is lost, what does an etcd snapshot bring back, and what must come from volume backups or Git instead?

level: middleimportance: must knowfreq 62%

answer

  1. three homes for cluster state
  2. objects versus bytes
  3. PVC object is not the data
  4. PKI and encryption key beside snapshot
  5. Git lacks generated Secrets

basics

~20 s

An etcd snapshot restores Kubernetes API objects (Deployments, Secrets, RBAC, PVC and PV objects, custom resources) but none of the bytes stored on persistent volumes. Volume data needs CSI snapshots or Velero; Git re-creates only what was committed.

solid answer

~40 s

There are three different kinds of state. **etcd** holds every API object the API server persists: workloads, `Secret`s, RBAC, CRDs and their custom resources, and the `PersistentVolume`/`PersistentVolumeClaim` objects that *describe* storage. A snapshot restores exactly that, at one instant. It does **not** hold the data on the volumes, node-local files such as `/etc/kubernetes/pki`, or cloud resources like load balancers. **Volume data** comes back from CSI `VolumeSnapshot`s, or from a tool like Velero that pairs object backups with volume snapshots. **Git** re-creates what the team committed, but not Secrets or objects that controllers generated at runtime, not imperatively created objects, and never data. A real recovery usually combines all three.

go deeper

for a junior

Remember the split: etcd keeps the list of Kubernetes objects, volumes keep application data, and Git keeps what the team wrote down. Backing up one does not back up the others.

for a middle

Explain what an etcd snapshot actually contains, including PV and PVC objects without their data, and what must be kept beside it: the PKI directory and the encryption-at-rest configuration.

for a senior

Show that you know a recovery combines all three sources, and name the gaps each one leaves, such as generated Secrets missing from Git or volumes whose PV objects point at deleted disks.

for a principal

Frame it as choosing the recovery unit: a whole cluster rewound to one instant, one namespace, or a rebuild from intent. Explain which failure each choice answers and who owns it.

## Three places cluster state lives When a Kubernetes cluster disappears, "restore it" can mean three different things, because the state lives in three different places: | State | Where it lives | What restores it | |---|---|---| | API objects (spec, status, metadata) | **etcd**, the cluster store behind `kube-apiserver` | an etcd snapshot, or an object-level backup tool | | Bytes written by workloads | **PersistentVolumes** on a storage backend | CSI `VolumeSnapshot`s, Velero with CSI snapshots or file-system backup | | Declared intent | **Git** (or another source of manifests) | re-applying the manifests, usually by a GitOps controller | A candidate who says "we back up etcd, so we're covered" has mixed up the first row with the other two. ## What an etcd snapshot contains **etcd** is the key-value store where `kube-apiserver` persists every object. A snapshot is a point-in-time copy of that store, so restoring it brings back: - every workload object: `Deployment`, `StatefulSet`, `Job`, `Pod` and so on; - `Secret` and `ConfigMap` objects, including Secrets that operators generated and that exist nowhere else; - RBAC objects, `Namespace`s, quotas, CRDs and the custom resources that use them; - `PersistentVolume` and `PersistentVolumeClaim` **objects**: the volume handle, capacity and binding, not the data; - `Node` objects and `status` fields, frozen at snapshot time, which is its own problem after a restore. It is also **all-or-nothing and cluster-wide**. You cannot restore one namespace from an etcd snapshot without rewinding every other namespace too. ## What the snapshot does not contain - **Volume contents.** Say a PDF-invoice renderer keeps its templates on a PVC. After an etcd restore, the PVC object exists and points at a volume handle. If that disk is gone, the pod is stuck on attach errors; nothing in etcd can recreate the templates. - **Node-local files** on control-plane machines: the cluster PKI under `/etc/kubernetes/pki` (CA, `sa.key`/`sa.pub`, front-proxy and etcd CAs), static pod manifests, kubelet config. - **The encryption-at-rest key file** passed to `kube-apiserver` with `--encryption-provider-config`. If Secrets were encrypted, the snapshot holds ciphertext, and without that file they are unreadable. - **External resources** that controllers created: cloud load balancers, DNS records, disks. These are referenced by objects but live outside the cluster. - **Container images**, which live in a registry. So an etcd backup procedure is really a bundle: the snapshot file, plus the PKI directory, plus the encryption configuration. All three must be stored off the cluster and access-controlled, because together they are the keys to every Secret. ## Where volume backups fit PV data needs its own mechanism. A **CSI VolumeSnapshot** asks the storage driver for a snapshot of one volume. **Velero** backs up API objects through the API server (by namespace, label or resource type) into object storage, and backs up volume data alongside them with CSI snapshots or file-system copies. It can restore just one team's namespace, which an etcd snapshot cannot. Without data movement, snapshots usually stay inside the same storage backend, so a lost region takes them with it. ## Where Git fits Re-applying manifests from Git into a fresh cluster is often the fastest path for **stateless** workloads, and it avoids restoring stale runtime state. But Git only holds what someone committed: - Secrets that a controller generated or someone created with `kubectl` are missing; - `status`, allocated `nodePort`s and similar runtime fields are gone; - a re-applied PVC gets a **new, empty** volume. ## Putting it together 1. Rebuild the cluster and restore PKI and encryption config (or accept new ones and re-issue credentials). 2. Bring back declared workloads from Git, or restore objects from an etcd snapshot or Velero backup. 3. Restore volume data from snapshots before the stateful pods start writing. 4. Check the gaps: Secrets, CRD instances and anything created outside Git. Interviewers ask this to find out whether a candidate knows that "backup" means three different things.

  • Why do Secrets need special handling when you store etcd snapshots?
    The snapshot contains every Secret. Without encryption at rest they are only base64-encoded, so anyone holding the file can read them. With encryption at rest, the snapshot is useless without the key file referenced by `--encryption-provider-config`. Either way, store the snapshot encrypted and access-controlled, and back up the encryption configuration separately, so a single leaked bucket does not expose both.
  • If every manifest is in Git, why keep etcd or Velero backups at all?
    Git does not hold state that only exists in the cluster: Secrets that operators generated, custom resources that teams created by hand, and `PersistentVolumeClaim` bindings to existing data. A pure Git rebuild also gives stateful workloads new, empty volumes. Object backups cover what never reached Git, and volume backups cover the data.
  • Does restoring an etcd snapshot restore Pod state such as in-memory data or running processes?
    No. The snapshot restores `Pod` objects, but containers are processes on nodes. After a restore, kubelets reconcile against the restored objects and start or stop containers to match. Anything held in memory is gone, and anything written to `emptyDir` or other node-local storage survives only if the node itself survived.

An etcd snapshot is like the index of a warehouse: it lists every shelf and what is supposed to be on it, but if the building burned down, the index does not bring the boxes back.

saying these in an interview costs you the question

  • An etcd snapshot includes the data stored on PersistentVolumes
  • Re-applying manifests from Git restores everything the cluster had
  • The snapshot alone is enough; PKI and encryption keys can be regenerated harmlessly
  • You can restore a single namespace from an etcd snapshot
  • Secrets in an etcd snapshot are safe because Kubernetes always encrypts them
open as a page

After losing a Kubernetes control plane, you restore last night's etcd snapshot into a replacement one. What breaks once kube-apiserver serves that state, and how do you handle it?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Restored state is old and bound to old identities. Reuse the original PKI and encryption keys, stop every API server before restoring, delete Node objects for vanished machines, restart controllers and kubelets, then reconcile drift caused by the rewind.

open as a page

You own disaster recovery for a 140-node Kubernetes cluster shared by 22 product teams. How do you split recovery between etcd snapshots, Velero and Git re-apply, and prove it works?

level: principalimportance: should knowfreq 38%

basics

~20 s

Match each failure to a recovery unit: rebuild plus Git re-apply for stateless workloads, Velero with CSI snapshots for namespaces with data or objects missing from Git, and etcd snapshots for rewinding the whole control plane. Prove it with timed restore drills.

open as a page

In a Velero restore of a Kubernetes namespace whose PVCs were backed up as CSI VolumeSnapshots, in what order are objects recreated, and how do PVCs get volumes again?

level: seniorimportance: nice to knowfreq 33%

basics

~20 s

Velero restores by priority: CRDs, namespaces, storage and snapshot classes, snapshot objects, PVs and PVCs, RBAC, Secrets and ConfigMaps, then Pods and controllers. CSI-backed PVs are not restored; each PVC gets a dataSource pointing at its VolumeSnapshot and is dynamically provisioned.

open as a page