A CronJob has produced nothing for two days and no kubectl command ever failed — what happened?

## The shape of the failure This is the canonical "blocked but invisible" outcome, and the batch path is where it hides longest. Nothing failed loudly because nothing a human did was rejected. The chain is: CronJob (admitted when someone applied it, possibly weeks ago) → Job, created by the cronjob controller on each schedule tick → Pod, created by the job controller. A rule matching only Pods lets the first two through and rejects the third. The requester on that rejected create is a control-plane controller, so the denial has nowhere to be printed. ## Why nobody noticed - **No Pod exists.** A rejected create produces no object, so `kubectl get pods` — including with `--show-all`-style filters or by label — returns nothing. There is no crash, no `Error`, no `ImagePullBackOff`, nothing to describe. Engineers looking for a failed pod find an empty list and assume the schedule did not fire. - **The Job looks empty rather than failed.** Its status typically shows no active, succeeded or failed pods at all — because "failed" counts pods that ran and terminated badly, and none ever ran. - **The evidence expires.** The create failure is an event on the Job, and clusters expire events after a short retention window, an hour by default in a stock configuration. Two days later the events are long gone. - **The evidence is also garbage-collected.** A CronJob keeps only a small number of finished Jobs by its history limits, so older Jobs — with whatever events they had — are deleted outright. - **The CronJob itself looks healthy.** Its last-schedule time keeps advancing, because scheduling worked; only the Pod create did not. ## How to confirm it quickly 1. Describe the most recent Job and read its events — a create failure quoting the rule's message is the smoking gun while it is still in the retention window. 2. Check the Job status for the tell: no pods in any state, rather than pods that failed. 3. Go to the API server audit trail and search for Pod create requests in that namespace made by the job controller's service account, and their response. This is the durable record when events have expired. 4. Check the admission engine's own denial records or metrics, if you run something that keeps them, filtered by namespace and resource. 5. Reproduce deterministically: create a Pod by hand from the job's pod template. As a human-submitted request, the same rule now returns the denial straight to your terminal with the message — which both confirms the cause and gives you the exact wording to hand to the owner. ## The wider lesson to give the interviewer The manifest fix is trivial. The real finding is that a guardrail was enforcing correctly for two days while the only evidence lived in an object nobody watches and a log that expires. Three durable improvements: - **Evaluate the same intent at the workload kinds as well**, so the next such change is rejected on the developer's `kubectl apply` or in CI, where a human is looking. Keep the Pod-level rule as the authoritative control. - **Alert on admission denials**, broken out by namespace and resource. A denial rate that goes from zero to steady in one namespace is exactly this failure, and it is visible within minutes. - **Alert on the workload, not just the gate.** A scheduled job that has not completed within its expected window should page regardless of the reason — that alert would have caught this on day one, and catches every other reason a batch job stops running too. ## A note on rollout sequencing Any new rule that will be evaluated against controller-created Pods should be watched at the workload level before it is allowed to reject. The failure mode above is not exotic; it is the default outcome of enforcing on an object no human submits, in a namespace whose owners were never told a rule was added.