A recurring job stops running entirely, with no error in the logs and the process still healthy. What common property of periodic scheduling explains this, and how do you defend against it?

## The rule that surprises everyone A periodic schedule is a loop the scheduler runs on your behalf, and its continuation condition is *the previous run finished normally*. Expressed as pseudocode: ``` run task if it completed normally: schedule the next occurrence else: record the failure and stop rescheduling ``` So a single escaping exception — on run number one or run number one million — terminates the entire recurrence. Not "skip this run." Not "retry next period." Terminate. The rationale is defensible: the scheduler has no idea whether the failure is transient or a permanent misconfiguration, and blindly repeating a task that throws instantly could produce an infinite error loop. So it stops and reports. The problem is *how* it reports. ## Why it is silent When you schedule a repeating task you typically get back a handle representing the task's eventual outcome. The escaping exception is stored **inside that handle**. It is not rethrown on any running thread, so: - No uncaught-exception handler fires, because the exception was caught by the scheduler. - No stack trace is printed, because nothing printed it. - The process stays perfectly healthy — this is one task among many. - Health checks pass, memory looks fine, CPU looks fine. The only way to see the failure is to inspect the handle, which requires a thread to ask for the result — and nobody does, because the task "runs forever" and there is no result to collect. The failure is therefore perfectly, structurally invisible. Teams commonly discover it days later from a downstream symptom: a cache never refreshed, a queue never drained, a report never generated, a certificate never renewed. ## The defenses ### 1. Never let anything escape the task body The only structural fix. Wrap the whole body in a catch that handles *every* throwable category the platform has, not just the ones you expect: ``` periodicTask(): try: doTheWork() catch anything as e: log.error("periodic task failed", e, context) metrics.failures++ # always return normally -> schedule survives ``` Two details people get wrong. First, catching only the "expected" error type leaves everything else escaping — and the errors that kill schedules in production are exactly the unexpected ones (a null dereference, a class-loading failure, an out-of-range index in code you did not think could fail). Second, the catch block itself must not throw; a logger that fails, or a metric emitter that throws, will escape and kill the schedule from inside the safety net. ### 2. Log with enough context to act on A schedule that survives failures but logs nothing is only a slower kind of invisible: the work is not happening and everything looks fine. Log the exception with the task identity, the attempt number, and any input that would let someone reproduce it. ### 3. Monitor the schedule, not the process This is the defense that catches everything, including the cases your wrapper misses (the task hung, the pool was shut down, the scheduler thread died, someone cancelled it). Emit either: - a **last-successful-run timestamp** per job, and alert when `now − last_success > k × period`, or - a **monotonically increasing success counter**, and alert when it stops increasing. The important property is that this is *liveness* monitoring: it detects absence of work. Almost all error monitoring detects presence of errors, which is exactly the wrong shape for this failure — there are no errors, only silence. Choose `k` to tolerate a couple of missed runs so ordinary jitter does not page anyone. ### 4. Bound each run A task that blocks forever — a network read with no timeout, a lock never released, a deadlock — never completes, so the next occurrence is never scheduled. The schedule is just as dead, and no exception was ever thrown, so the wrapper does not help. Give each run its own timeout or deadline so a hung attempt is abandoned and the recurrence continues. On a scheduler with a small worker count, one hung task can also starve *other* schedules that share those workers. ### 5. Distinguish transient from permanent Swallowing every error forever is not always right either. If a job fails identically for hours, something needs to escalate. A good pattern is to keep the schedule alive but track consecutive failures, log at increasing severity, and alert after a threshold — you get both liveness and honesty. ## The mental model to carry away There are three distinct states a recurring job can be in, and only the first is healthy: 1. **Running and succeeding** — the counter advances. 2. **Running and failing** — errors appear in logs; visible, fixable. 3. **Not running at all** — total silence, everything green. State 3 is the one this question is about, and the only way to detect it is to monitor for the absence of successful runs.