A recurring job is configured to run every 30 seconds, but under load individual runs start taking 90 seconds. Describe what a typical scheduler does in that situation and how you would make the job behave sensibly.

## The situation Period P = 30 s, execution E = 90 s. The schedule asks for something arithmetically impossible: three runs' worth of due times pass during every single run. ## What schedulers actually do **Serialization is the norm.** Nearly every general-purpose scheduler treats a periodic task as a single logical job and will not start occurrence *n+1* while occurrence *n* is running. What you get is: - occurrence 2 becomes due at t=30 while run 1 is still going, - occurrence 3 due at t=60, occurrence 4 at t=90 — all overdue, - run 1 finishes at t=90, and the next run starts *immediately*. The steady state is back-to-back execution at the execution time: a run every 90 s with zero idle. Two harms follow. First, the configured cadence is silently gone; anything downstream expecting a 30-second refresh is now three times stale, with no error anywhere. Second — the dangerous one — **the job's pressure on the system goes up exactly when the system is struggling.** Normally the job worked for 30 s out of every 90 s at most; now it works 100% of the time against the same slow database or API, which makes it slower still. That is a positive feedback loop, and it is how a harmless maintenance job becomes the proximate cause of an outage. A second question is what happens to the *missed* occurrences. Some schedulers coalesce them (one immediate run, then re-anchor); others attempt to catch up by replaying each. Catch-up replay is worse: it fires a burst of runs at a system that just fell behind. **Overlapping is possible and worse.** Some application-level and distributed schedulers will start a new instance regardless. Then you have concurrent copies of a job usually written assuming it is alone: rows processed twice, lock contention or deadlock between instances, memory growing with the number of live instances, and — because each instance is slow — an ever-growing set of them until resources run out. ## Making it behave ### 1. Measure the run, not just the schedule Export duration percentiles per job and alert when p99 duration exceeds, say, half the period. The transition from "comfortable" to "continuous execution" is gradual in reality and invisible without this metric. Duration approaching the period is the single best leading indicator. ### 2. Bound each run Give every run a deadline or timeout. Without one, a run that blocks forever suspends the schedule permanently — indistinguishable, from the outside, from a cancelled schedule. With one, a pathological attempt is abandoned and the cadence resumes. ### 3. Prefer a gap-based interval when duration is variable An interval measured from the previous run's *end* makes the period stretch to E + P automatically. The target always gets P of quiet, no matter how slow the job becomes. This is self-limiting by construction and is the right default for anything polling or scanning a shared resource. You give up cadence guarantees, which is the correct trade when the cadence was already unachievable. ### 4. Decide the missed-run policy explicitly For most maintenance work, **skip** is right: if three refresh windows passed, run once now and re-anchor to the next due time — there is no value in doing the same refresh three times. **Catch-up** is only right when each occurrence corresponds to distinct work that must not be lost (per-hour billing aggregation, for example), and even then it should be rate-limited so recovery does not become a thundering burst. Make this a stated policy per job, not an accident of the library's default. ### 5. Guard against overlap explicitly Do not rely on the scheduler's serialization if you might ever move to a different scheduler or run more than one instance of the service. Use a guard the job itself owns: a flag or lease checked at entry that causes an immediate no-op return (and increments a "skipped because still running" counter) if a previous run is in flight. In a multi-node deployment this must be a shared lease with an expiry, not an in-process flag — otherwise every node runs its own copy. ### 6. Make the job idempotent If running twice concurrently or replaying a missed occurrence is harmless, every other problem here becomes survivable. Idempotence is the property that turns a scheduling correctness problem into a scheduling efficiency problem. ### 7. Fix the size mismatch All of the above manages symptoms. If a 30-second job genuinely takes 90 seconds, the work does not fit the window. Chunk it (process a bounded batch per run and continue next time), shard it across instances by key range, or lengthen the period to match reality. Choosing a period without measuring the work is the original defect. ## The summary an interviewer wants Overrun does not produce an error — it produces silent loss of cadence, and, with fixed-rate semantics, a job that consumes resources continuously at the worst possible time. The engineering answer is: measure duration, bound each run, prefer gap-based intervals for variable work, define the missed-run policy deliberately, guard overlap with a lease, and ultimately make the work fit the window.