After raising a Kubernetes PersistentVolumeClaim's spec.resources.requests.storage, why can kubectl get pvc still show the old capacity, and what completes the resize?
answer
- two actors, two phases
- backend first, filesystem second
- CAPACITY column moves last
- a condition that waits for a mount
- online versus restart-required driver
basics
~20 sExpansion has two phases. The storage backend grows the volume first, then the kubelet grows the filesystem on the node. status.capacity, which kubectl shows, changes only after both finish. FileSystemResizePending means the node step is waiting for the volume to be mounted.
solid answer
~40 sEditing `spec.resources.requests.storage` records the desired size; the `CAPACITY` column shows `status.capacity`, which changes last. First comes **controller expansion**: for CSI volumes the driver's external-resizer calls the storage backend, the PV's capacity grows, and the PVC carries a `Resizing` condition. Next comes **node expansion**: the kubelet on the node that mounts the volume calls the driver to grow the filesystem. Until it does, the PVC shows `FileSystemResizePending` and `status.allocatedResourceStatuses.storage` shows `NodeResizePending`. If the driver supports **online** expansion, the kubelet finishes this while the pod keeps running. Otherwise the pod must be restarted so the volume is mounted again. Only after both phases does `status.capacity` show the new size. Failures surface as `ControllerResizeError` or `NodeResizeError` conditions.
code
bash · 2 lineskubectl get pvc checkout-db-data -n checkout \
-o jsonpath='{range .status.conditions[*]}{.type}{": "}{.message}{"\n"}{end}'go deeper
Remember that expansion is backend growth followed by filesystem growth, and that kubectl's capacity column only updates at the end.
Explain which component does each phase, what Resizing, FileSystemResizePending and the allocatedResourceStatuses values mean, and when a pod restart is needed.
Show how you diagnose a stalled resize from conditions and events, and why you check online-expansion support and quota before the disk fills.
Consider how the platform makes expansion safe by default: classes that allow it, alerts well before full, and drivers chosen for online growth.
## The symptom It is 3am, the checkout database's PVC `checkout-db-data` on a 3-node kubeadm cluster is at 97% of 80Gi, and you patch the request to 140Gi. `kubectl get pvc` still prints `80Gi`. Nothing is broken yet. The `CAPACITY` column is `status.capacity`, the **last** field to change, and expansion is a two-phase pipeline. ## Phase 1: controller expansion 1. The API server admits the change only if the claim is `Bound` and its StorageClass has `allowVolumeExpansion: true`. Otherwise you get `only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize`. 2. The expand controller in kube-controller-manager sees that the request exceeds the capacity. For CSI volumes it hands the work to the driver's resizer and records it in the annotation `volume.kubernetes.io/storage-resizer`. 3. The resizer asks the storage backend to grow the volume. Meanwhile the PVC shows a `Resizing` condition, `status.allocatedResources.storage` holds the target size, and `status.allocatedResourceStatuses.storage` is `ControllerResizeInProgress`. 4. When the backend finishes, the PV's `spec.capacity` is updated. If the filesystem must still grow, the PVC gets `FileSystemResizePending` and the resource status becomes `NodeResizePending`. ## Phase 2: node expansion The block device is now bigger, but the filesystem on it is not. The **kubelet** on the node that mounts the volume performs this step through the CSI node service: - **Online expansion.** If the driver supports growing a mounted volume, the kubelet does it while the checkout pod keeps serving, even during a 3,400 requests-per-second burst. The status moves to `NodeResizeInProgress`, then the conditions clear and `status.capacity` becomes `140Gi`. - **Offline expansion.** If the driver cannot, the condition message reads `Waiting for user to (re-)start a pod to finish file system resize of volume on node.` You delete the pod, the controller that owns it recreates it, the volume is mounted again, and the kubelet grows the filesystem during that mount. - **No pod at all.** A claim that no pod mounts stays in `FileSystemResizePending` until some pod mounts it. That is expected, not a fault. | Signal on the PVC | Meaning | |---|---| | `Resizing` condition / `ControllerResizeInProgress` | backend growth under way | | `FileSystemResizePending` / `NodeResizePending` | backend done, filesystem growth waiting for the kubelet | | `NodeResizeInProgress` | kubelet is growing the filesystem | | `ControllerResizeError` or `NodeResizeError` condition | the attempt failed; read the message and events | | `ControllerResizeInfeasible` / `NodeResizeInfeasible` | the failure is terminal for this size; lower the request to recover | | no conditions, `status.capacity` = request | finished | ## Reading it in practice ```bash kubectl patch pvc checkout-db-data -n checkout --type merge \ -p '{"spec":{"resources":{"requests":{"storage":"140Gi"}}}}' kubectl get pvc checkout-db-data -n checkout \ -o jsonpath='{.status.capacity.storage}{" "}{.status.allocatedResourceStatuses}{"\n"}' kubectl describe pvc checkout-db-data -n checkout # conditions and events ``` The fastest diagnosis is `kubectl describe pvc`: its conditions tell you which phase you are in, and its events carry the resizer's or kubelet's error text. ## Rules that shape the procedure - **Growth only.** A request below the previous value is rejected. The single exception, lowering a failed request while staying above `status.capacity`, exists to recover from failed expansions and never shrinks a volume. - **The request is what quota sees.** ResourceQuota charges the larger of `allocatedResources` and the request, so a pending resize already counts against `requests.storage`. - **Raw block volumes** (`volumeMode: Block`) have no filesystem for the kubelet to grow; the application sees a larger device. - **Check the application too.** Some databases cap their own data-file size or cache free space at startup, so a grown filesystem may still need a configuration change or a restart. ## Why interviewers ask The question separates people who have edited a number in YAML from people who have watched an expansion stall. The strong answer names both phases, knows that `status.capacity` moves last, reads `FileSystemResizePending` as "waiting for a mount", not "broken", and knows whether the driver in use supports online expansion before the disk is full.
- The PVC shows FileSystemResizePending but no pod uses the claim. Is the resize stuck?No. The backend has already grown the volume, and the filesystem step runs only when the kubelet mounts the volume. The condition stays until a pod that uses the claim starts. Once the kubelet mounts the volume, it grows the filesystem, the condition clears and `status.capacity` shows the new size. Nothing needs to be retried.
- How do you tell whether the resize needs a pod restart?Look at the PVC after the controller phase. If `FileSystemResizePending` appears and its message asks you to (re-)start a pod while the pod is running, the driver does not support online expansion, so restart the pod through its owning controller. If the driver supports online expansion, the kubelet clears the condition by itself within a sync or two.
- What does status.allocatedResources tell you that spec and status.capacity do not?It records the size the resize machinery is working towards, which can differ from both the current request and the finished capacity. ResourceQuota also charges the larger of `allocatedResources` and the request, which is why a stuck or mistyped resize still consumes `requests.storage` quota.
saying these in an interview costs you the question
- The CAPACITY column changes the moment you edit the request.
- FileSystemResizePending means the expansion has failed.
- Every PVC expansion requires deleting and recreating the pod.
- The kube-scheduler performs the filesystem resize on the node.
- Growing the PV object's capacity field by hand resizes the disk.