A Kubernetes CronJob stopped firing after the cluster control plane was unavailable for several hours, and the controller logs mention too many missed start times. Explain the mechanism behind that, and what role `startingDeadlineSeconds` plays.

## How the controller decides to fire The CronJob controller does not hold a timer per object. It reconciles: on each pass (roughly every 10 seconds) it takes the CronJob's `schedule`, its `.status.lastScheduleTime`, and the current time, and enumerates the schedule points that fall between them. For each such point it decides whether to create a Job. That design is what makes the controller stateless and restartable — but it means a gap in reconciliation (an unreachable API server, a crashed or evicted controller-manager, an etcd outage, a suspended CronJob later resumed, a clock jump) leaves a **backlog** of schedule points to consider on the next pass. ## The 100-missed-schedules guard Enumerating that backlog is unbounded work: a `* * * * *` schedule that has been dark for two days is nearly 3,000 points. Rather than iterate indefinitely — or, worse, fire thousands of Jobs in a burst — the controller stops counting once it exceeds **100** missed start times, refuses to schedule, and records an event and log line: `Cannot determine if job needs to be started: too many missed start times`. The crucial operational fact is that this state is **sticky**. The CronJob does not heal itself when the control plane comes back, because `lastScheduleTime` is still far in the past and the backlog is still over 100 on every subsequent pass. `kubectl get cronjob` shows a healthy-looking object with an old `LAST SCHEDULE`, and nothing fires. Teams typically discover it days later. Recovery is manual: either update `.status.lastScheduleTime` to a recent value, or — far more commonly and safely — recreate the CronJob (delete and re-apply), which resets the reference point. Then set `startingDeadlineSeconds` so it cannot recur. ## What startingDeadlineSeconds actually does The field has two effects, and both matter here. **Lateness bound.** If a schedule point was missed, the controller starts a Job for it only if the current time is within `startingDeadlineSeconds` of that point. Otherwise the occurrence is abandoned and counted in `.status` as missed. This is what you want for time-sensitive work: a report due at 02:00 is usually worthless if it starts at 09:00. **Lookback bound.** When counting missed schedules, the controller only considers points inside that window instead of walking all the way back to `lastScheduleTime`. That is what keeps the count below 100 and prevents the lock-up entirely. With the field **unset**, there is no lateness bound and no lookback bound — the controller enumerates everything since the last run, which is precisely the path into the 100-schedule guard. ## Choosing a value Set it to a small multiple of the schedule interval, long enough to survive a brief control-plane hiccup, short enough that a stale run never fires: - every 5 minutes → 120–300 seconds - hourly → 600–1800 seconds - daily → 3600 seconds or so, if a late start is acceptable at all Avoid values below about 10 seconds. The controller's poll interval is around 10 seconds, so a deadline shorter than that can expire between passes and the occurrence is skipped even with a perfectly healthy cluster. ## What happens after recovery With a sane `startingDeadlineSeconds`, an outage produces at most **one** catch-up run — the most recent missed occurrence, if it is still within the deadline. Everything older is abandoned. That is almost always the behaviour you want; the alternative (a thundering herd of backlogged Jobs all starting at once) can take out the database the jobs talk to. If you genuinely need every missed occurrence executed, CronJob is the wrong tool — that is a workflow engine or a durable queue with its own backlog semantics. ## Detecting the failure Because the object looks healthy, alert on **outcome**, not on object state: - age of `.status.lastSuccessfulTime` exceeding a few intervals, - age of `.status.lastScheduleTime`, - the "too many missed start times" event on the CronJob, - a heartbeat written by the job payload itself into a system you already monitor. The last one is the most robust, because it survives the CronJob object being deleted, suspended, or misconfigured, and it also catches jobs that "run" but do nothing useful. ## Related causes of the same symptom The same lock-up appears without any outage if a CronJob is left `suspend: true` for a long time and then resumed, or if the controller-manager's clock jumps backwards or forwards significantly. And a CronJob whose Jobs pile up under `concurrencyPolicy: Forbid` with a hung run stops firing for an entirely different reason — worth distinguishing during diagnosis, since the remedy differs.