skip to content

A Kubernetes PVC is mistakenly resized from 80Gi to 8000Gi and the expansion fails. How do you recover, given that claims cannot shrink?

level: seniorimportance: nice to knowfreq 28%

answer

  1. no shrink, but a way back
  2. strictly above status.capacity
  3. feature gate that went GA in 1.34
  4. quota counts the larger value
  5. grown too far means migrate

basics

~20 s

Lower spec.resources.requests.storage to a realistic size that is still above status.capacity, which RecoverVolumeExpansionFailure allows (GA since 1.34). The resizer then retries the smaller target. A true shrink still means a new, smaller claim and a data copy.

solid answer

~40 s

A bound claim normally rejects a smaller request, but the `RecoverVolumeExpansionFailure` behavior (GA and locked on since v1.34) lets you lower the request to any value **greater than** `status.capacity`. Anything at or below it fails with `field can not be less than status.capacity`. So for a claim at 80Gi that failed on 8000Gi, set the request to, say, 140Gi. The resizer drops the impossible target and retries the new one. Check `kubectl describe pvc`: `ControllerResizeError` and a `ControllerResizeInfeasible` status mark the failure, and `status.allocatedResources` shows the target. That matters because ResourceQuota charges the larger of `allocatedResources` and the request until it drops. If the backend already grew the volume, nothing gives the space back; a real shrink means provisioning a smaller claim and copying the data.

code

bash · 1 line
bash
kubectl get pvc checkout-db-data -n checkout -o jsonpath='{.status.capacity.storage}{" "}{.status.allocatedResources.storage}{" "}{.status.allocatedResourceStatuses.storage}{"\n"}'

go deeper

for a junior

Remember that a PVC cannot shrink, and that a mistyped expansion is fixed by lowering the request to a size still above the current capacity.

for a middle

Explain the validation rule: requests may drop only while they stay above status.capacity, and allocatedResources tracks the target in the meantime.

for a senior

Show the whole recovery: read the conditions, pick a valid size, watch both phases, clear the quota effect, and know when only a migration will do.

for a principal

Decide which guardrails stop a 100x typo: quotas, admission caps on claim size, and reviewed runbooks, weighed against friction for legitimate growth.

## The incident During a 3,400 requests-per-second checkout burst, the checkout database's claim `checkout-db-data` (80Gi, on a 3-node kubeadm cluster backed by an on-prem array) is nearly full. An engineer patches the request to `8000Gi` instead of `140Gi`. The array cannot supply that, so expansion fails. The obvious fix, setting the request back to 80Gi or 140Gi, bumps into the rule that PVCs never shrink. ## What the API allows The PVC update validation in v1.37 has two branches: 1. With `RecoverVolumeExpansionFailure` disabled, any request below the previous request is rejected: `field can not be less than previous value`. 2. With it enabled (the default, and locked on since v1.34), a lower request is accepted **as long as it stays strictly above** `status.capacity`. Otherwise: `field can not be less than status.capacity`. So the recovery window is `status.capacity < new request < failed request`. With capacity at 80Gi, `140Gi` is valid; `80Gi` is not, because the rule is strictly greater. The feature exists only to **back out a failed expansion**. It never shrinks a volume. Its own validation comment says Kubernetes does not support volume shrinking. ## Recovery procedure 1. **Confirm the failure mode.** `kubectl describe pvc checkout-db-data -n checkout` shows a `ControllerResizeError` condition with the backend's message. `status.allocatedResourceStatuses.storage` shows `ControllerResizeInfeasible` when the driver reports the size as impossible; transient errors keep being retried instead. 2. **Lower the request** to the size you meant, still above capacity: ```bash kubectl patch pvc checkout-db-data -n checkout --type merge \ -p '{"spec":{"resources":{"requests":{"storage":"140Gi"}}}}' ``` 3. **Watch it converge.** The resizer retries with the new target, then the node phase grows the filesystem, and `status.capacity` becomes `140Gi`. 4. **Check quota.** ResourceQuota counts the larger of `status.allocatedResources` and the request. `allocatedResources` drops only when no expansion is in progress and the real capacity is at or below the new request. Until then, the namespace's `requests.storage` may look exhausted and block unrelated claims. ## When the volume already grew If the backend **succeeded**, perhaps growing to something larger than intended, nothing Kubernetes offers returns the space. Lowering the request cannot reach below the new `status.capacity`. Your options: - **Live with it** if cost is acceptable. - **Migrate to a smaller claim.** Provision a new PVC of the right size, copy the data with the database's own dump-and-restore or replication, and switch the workload over. A **clone** or **snapshot restore** will not help: both require a request at least as large as the source. ## How it used to be done Before this recovery path existed, the documented workaround was manual surgery. You set the PV's `persistentVolumeReclaimPolicy` to `Retain`, deleted the PVC, removed the PV's `claimRef`, and recreated a PVC with a smaller request pre-bound to that PV. It still appears in older runbooks. It is risky, needs downtime, and is unnecessary on current releases. | Situation | Action | |---|---| | expansion failed, capacity unchanged | lower request to a value above `status.capacity` | | lowered request equals capacity | rejected; pick a value strictly larger | | backend grew too far | no in-place fix; migrate to a smaller PVC | | quota looks full after the typo | wait for `allocatedResources` to drop after recovery | ## Guardrails worth adding - A **ResourceQuota** with a sensible `requests.storage` catches a 100x typo at admission, before any backend call. - An admission policy can cap the size a single claim may request. - Runbooks should use `kubectl patch` with a reviewed value, not ad-hoc `kubectl edit` during an incident.

  • Why can't you clone the 140Gi of real data into a 100Gi claim to shrink it?
    A PVC clone or snapshot restore copies the source at the block level. It must request at least the source's size (for a snapshot, its `restoreSize`), so a smaller target is refused. Shrinking means a logical copy: create a small empty claim, move the data with the database's own tools, then cut the workload over.
  • Why did an unrelated team's new PVC fail with a quota error right after the typo?
    ResourceQuota charges each claim the larger of `status.allocatedResources` and its request. While the 8000Gi target is recorded, the namespace's `requests.storage` looks consumed. Lowering the request, and waiting until no expansion is in progress, lets `allocatedResources` drop and frees the quota.

saying these in an interview costs you the question

  • You can set the request back to exactly the old capacity.
  • Lowering the request shrinks the volume on the storage backend.
  • The only fix is deleting the PVC and restoring from backup.
  • A failed expansion never affects the namespace's storage quota.
  • Editing the PV's capacity field makes the claim smaller.