skip to content

A team takes a nightly Kubernetes VolumeSnapshot of a checkout database's PersistentVolumeClaim. Why is that, on its own, not a real database backup?

level: juniorimportance: should knowfreq 46%

answer

  1. where does the copy physically live
  2. power-cut image, not a clean one
  3. two volumes, two instants
  4. deletionPolicy follows the object
  5. no schedule, no restore drill

basics

~20 s

A VolumeSnapshot normally lives on the same storage system as the volume, is only crash-consistent, and is deleted along with its Kubernetes object under the Delete policy. A backup needs application consistency, a separate failure domain, independent retention and a tested restore.

solid answer

~40 s

A `VolumeSnapshot` asks the CSI driver for a point-in-time copy. The copy is recorded in a `VolumeSnapshotContent` whose `snapshotHandle` points into the **same storage system** as the volume, so losing that system also loses the snapshots. The copy is **crash-consistent**: nobody flushes or freezes the database, so the database must run crash recovery on restore, and a database spread across two PVCs is not captured atomically. The physical snapshot's lifetime follows Kubernetes objects: with `deletionPolicy: Delete` on the `VolumeSnapshotClass`, deleting the `VolumeSnapshot` or its namespace destroys it. A real backup adds an application-consistent capture (for example a database-native dump or a freeze), a copy in a separate failure domain, retention that does not depend on the cluster, and restores that are regularly tested.

code

yaml · 6 lines
yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: array-snapclass-retain
driver: csi.array.example.com
deletionPolicy: Retain

go deeper

for a junior

Remember the three gaps: same storage system, crash-consistent only, and lifetime tied to the VolumeSnapshot object through deletionPolicy. Say what a real backup adds.

for a middle

Explain the object split: namespaced VolumeSnapshot, cluster-scoped VolumeSnapshotContent and VolumeSnapshotClass, and how readyToUse and deletionPolicy behave.

for a senior

Show how you would tier recovery: snapshots as fast local restore points, an application-consistent copy in a separate failure domain, and restore drills that prove both.

for a principal

Frame the tradeoff between restore speed and failure-domain independence, and set a policy for Retain versus Delete classes and who owns pruning across teams.

## What a VolumeSnapshot actually is Kubernetes models volume snapshots with three API objects in the `snapshot.storage.k8s.io/v1` group. They are CRDs served by a separately installed **snapshot-controller** plus the driver's `csi-snapshotter` sidecar, not by the core API server. That matters on a kubeadm cluster, which ships none of them by default. | Object | Scope | Role | |---|---|---| | `VolumeSnapshotClass` | cluster | names the CSI `driver`, its `parameters` and the `deletionPolicy` | | `VolumeSnapshot` | namespace | the user's request: `spec.source.persistentVolumeClaimName` plus `volumeSnapshotClassName` | | `VolumeSnapshotContent` | cluster | the record of the real snapshot, carrying the driver's `snapshotHandle` | Once the driver finishes, the `VolumeSnapshot` reports `status.readyToUse: true`, a `restoreSize` and `boundVolumeSnapshotContentName`. That is the whole contract: the storage system holds a point-in-time image of one volume, and Kubernetes holds a pointer to it. ## Why that is not a backup Take a ticket-booking checkout whose database sits on a PVC on a 3-node on-prem kubeadm cluster. A CronJob creates a `VolumeSnapshot` every night. Four gaps remain: 1. **Same failure domain.** The snapshot normally lives inside the same storage array or storage cluster as the volume. If the array dies, or someone deletes the storage pool, the volume and every snapshot of it go together. A backup must survive the loss of the thing it protects. 2. **Crash consistency only.** CSI `CreateSnapshot` does not ask the database to flush or pause. The image looks like the moment after a power cut. A well-behaved database replays its log and recovers, but writes still in memory during a 3,400 requests-per-second checkout burst are not in the image. If data and write-ahead log sit on **two** PVCs, two separate snapshots are taken at slightly different instants. The `VolumeGroupSnapshot` API (`groupsnapshot.storage.k8s.io`, still a beta API) exists to close exactly that gap. 3. **Lifetime tied to cluster objects.** With `deletionPolicy: Delete`, deleting the `VolumeSnapshot`, or the namespace containing it, deletes the `VolumeSnapshotContent` and the physical snapshot. A `kubectl delete namespace checkout` becomes a delete of the backups too. `Retain` keeps the content object and the physical snapshot, but then someone has to own cleanup. 4. **No schedule, retention or verification.** Kubernetes has no built-in snapshot schedule or pruning. Whatever creates the objects also has to delete old ones, and a snapshot nobody has ever restored is only a hope. ## What turns it into a backup - **Application consistency.** Take a database-native dump, or quiesce the database (flush and freeze) around the snapshot, or use a group snapshot when data spans several volumes. - **A separate copy.** Export the data to storage outside the array and outside the cluster. Whole-cluster backup tools that move CSI snapshot data off-site belong to the cluster backup-and-restore topic. The point here is only that *something* must copy the data out. - **Independent retention.** Pruning must not depend on the namespace or the `VolumeSnapshot` object staying alive. Consider `Retain` for anything you would regret losing. - **Restore drills.** Regularly create a PVC from a snapshot (`spec.dataSource` of kind `VolumeSnapshot`) in a scratch namespace, start the database against it and check the data. ## Where snapshots still earn their place Snapshots are excellent **fast local restore points**: before a schema migration, before an upgrade of the checkout service, or to clone production-shaped data for a test. They restore in seconds to minutes because no data crosses the network. Treat them as the first tier of a recovery plan, never the only one. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: checkout-db-pre-migration namespace: checkout spec: volumeSnapshotClassName: array-snapclass source: persistentVolumeClaimName: checkout-db-data ``` ## Quick checks an interviewer likes - `kubectl get volumesnapshot -n checkout` shows `READYTOUSE` and the restore size. A snapshot that never becomes ready is not a restore point. - `kubectl get volumesnapshotclass -o yaml` reveals the `deletionPolicy` that decides whether your snapshots outlive their objects. - The annotation `snapshot.storage.kubernetes.io/is-default-class: "true"` marks the class used when a `VolumeSnapshot` omits `volumeSnapshotClassName`.

  • What does deletionPolicy Retain change, and what does it not fix?
    With `Retain`, deleting the `VolumeSnapshot` leaves the cluster-scoped `VolumeSnapshotContent` and the physical snapshot in place, so a namespace deletion no longer destroys restore points. The snapshot still sits on the same storage system, is still only crash-consistent, and now needs someone to prune it. Otherwise orphaned snapshots pile up on the array.
  • How do you verify a nightly VolumeSnapshot is actually restorable?
    Wait for `status.readyToUse: true`. Then, in a scratch namespace, create a PVC whose `spec.dataSource` names the snapshot (`kind: VolumeSnapshot`, `apiGroup: snapshot.storage.k8s.io`) with a request of at least `restoreSize`. Start the database against it, let crash recovery run, and check recent checkout rows. A VolumeSnapshot is namespaced, so the drill PVC must live in the snapshot's namespace, or you must make the snapshot available there first.

A VolumeSnapshot is like a photocopy kept in the same filing cabinet as the original: quick to grab, but a fire in the cabinet takes both.

saying these in an interview costs you the question

  • A VolumeSnapshot copies the data off the storage system automatically.
  • Snapshots are application-consistent because Kubernetes pauses the pod first.
  • Deleting the namespace never touches the underlying snapshots.
  • Kubernetes rotates and prunes old VolumeSnapshots on a built-in schedule.
  • Two PVCs snapshotted one after another form one consistent restore point.