An eleven-month-old StatefulSet cannot scale up eight weeks after a PVC label policy shipped — what happened?
answer
- The rule matched the child, not the parent
- Scale-up creates a new claim
- Old replicas were never re-judged
- The template predates enforcement
- Claim templates cannot be patched later
basics
~20 sThe StatefulSet's volume claim template predates the policy and lacks the required label. Its existing replicas were never re-evaluated, but each new replica makes the controller create a fresh PVC, and that create is denied at admission.
solid answer
~40 sThe policy matches PersistentVolumeClaims, not StatefulSets, so the StatefulSet object itself was never re-submitted and never judged — it has sat unchanged for eleven months. Its existing replicas run on PVCs created before the rule existed. Scaling up makes the StatefulSet controller create a new claim from `volumeClaimTemplates`, and that is an ordinary CREATE request that admission denies for the missing label. The pod stays Pending because its volume never appears; existing replicas are unaffected, which is why nothing looked wrong for eight weeks. The fix is not a quick patch: `volumeClaimTemplates` is immutable on an existing StatefulSet, so correcting the template means recreating the StatefulSet, orphaning its pods to avoid downtime. Right now, unblock with a narrow, time-boxed exception, then schedule the template fix.
go deeper
Know that a rule about PersistentVolumeClaims fires when a claim is created, whoever creates it — including a controller acting on behalf of a workload — and not when the workload was originally deployed.
Be able to trace the chain: scale-up, controller creates a claim from the stored template, admission denies it, pod stays Pending, existing replicas unaffected. Explain why the failure was invisible until then.
Show incident judgment: narrow time-boxed exception to unblock, then a planned template fix, plus the pre-enforcement inventory of spawning templates that would have prevented the surprise entirely.
Own the rollout policy that makes latent blast radius everyone's default consideration: before a rule goes to enforce, someone must have enumerated the stored templates that will spawn objects it matches.
## The shape of the failure A rule requiring a backup-retention label on PersistentVolumeClaims shipped eight weeks ago. Every PVC created since has the label. Then at 02:00 a StatefulSet that nobody has edited in eleven months is scaled up during a traffic spike, and the new replica never becomes ready. The chain is: 1. The rule matches `persistentvolumeclaims`. It has never seen this StatefulSet, because a rule only evaluates the resource it matches, and the StatefulSet was not written since the rule shipped. 2. The StatefulSet's `volumeClaimTemplates` were authored eleven months ago and carry no such label — nobody knew to add it. 3. Scaling up makes the **StatefulSet controller** create a new PVC from that template, through the ordinary API, as an ordinary CREATE. 4. Admission denies it. The controller retries and backs off; the pod has no bound claim and stays Pending; the replica never joins. 5. The already-running replicas keep working, because their PVCs were created before enforcement began and nothing re-evaluates them. So the policy did exactly what it was written to do, and its blast radius was latent for eight weeks until an unrelated capacity event triggered it. ## Why this class of bug is easy to miss When you enforce on a **child** object that a parent template spawns, the population at risk is not "objects created since the rule shipped". It is **every stored template that will ever spawn another child** — every StatefulSet, every controller that manufactures objects from a spec you have not looked at. Those templates were admitted long ago, are not matched by your rule, and will fail on their next expansion event: a scale-up, a namespace rebuild, a restore into a fresh cluster, a disaster-recovery drill. The worst part is the timing: expansion events cluster around load spikes and recoveries, so the denial arrives at the moment you least want a new failure mode. ## Handling it on the night First, decide whether the policy or the workload is the emergency. It is not: the workload is. Unblock with the narrowest, shortest-lived exception you can express — this one namespace or this one claim-name prefix, with an expiry and a ticket — rather than disabling the rule cluster-wide or setting the whole webhook aside. A blanket bypass converts one stalled scale-up into an unmeasured hole in every namespace. Then record it. An exception without an expiry and an owner is a silent permanent hole; the reason to time-box it is that the underlying template is still broken. ## Why the fix is not a patch On an existing StatefulSet, `volumeClaimTemplates` is **immutable** — the API server rejects updates to it, as it does to most of the StatefulSet spec outside replicas, the pod template, update strategy and a few related fields. So you cannot simply add a label to the template. The realistic path is to delete the StatefulSet non-cascading, so its pods and PVCs are orphaned and keep running, then recreate the StatefulSet with a corrected template so it adopts them. That is a planned change with a rehearsal, not something to attempt at 02:00 under load. ## The lesson to carry forward Before enabling a rule on a resource, ask **who creates that resource and from what**. If the answer includes controllers expanding stored templates, then your legacy population includes those templates, and you should enumerate them and fix them *before* enforcement rather than discovering them one incident at a time. That inventory pass is also what tells you the real size of the problem — the gate's own numbers cannot, because the objects it has never seen do not appear in them. ## What a strong answer sounds like It separates three things: the rule is correct, the workload's template is non-conforming, and the discovery mechanism was an outage. It fixes the third — pre-enforcement inventory of the templates that spawn matched objects — not just the first two.
- What do you do in the first ten minutes, before anything is fixed properly?Confirm the denial is the cause by reading the failed PVC create and the controller's events, then grant the narrowest possible exception — that namespace or that claim name, with an expiry and a ticket — so the replica can come up. Do not disable the rule cluster-wide; one stalled scale-up does not justify removing enforcement from every other namespace. Then hand off the template fix as planned work.
- How would you have found this before it broke?By enumerating stored objects that spawn matched children and evaluating their templates against the rule ahead of enforcement — StatefulSets' volume claim templates here, and by extension anything else that manufactures objects from a stored spec. That pass gives you the real legacy count, which the gate's telemetry cannot, and lets you fix templates during a change window instead of during a scale-up.
- Would enforcing on the StatefulSet resource instead have avoided this?It would have moved the failure earlier and made it more visible, since the workload author would have been blocked at submit time rather than a controller failing later. But it does not help for the eleven-month-old object either, because it was never re-submitted. Enforcing at both levels catches new templates at authoring time and still needs an inventory pass for the ones already stored.
The building's new door policy only checks new arrivals, so nothing breaks until an old resident tries to bring in one more guest, using an invitation printed a year ago.
saying these in an interview costs you the question
- Assumes the policy re-evaluated the StatefulSet and blocked the scale
- Blames the running replicas rather than the new claim
- Disables the rule cluster-wide to unblock one workload
- Thinks the claim template can simply be patched in place
- Says the eight quiet weeks prove the estate was compliant