Why does a nightly billing job scheduled by a local cron entry on each of five identical nodes run incorrectly?
answer
- each machine has its own scheduler
- decision is local, effect is global
- pinning one node flips the failure
- shared claim per job and tick
- idempotent per tick as backstop
basics
~20 sEach node's local scheduler fires on its own, so the job runs five times per tick and customers can be billed five times. The fire decision must be made once for the cluster, by an elected runner or a shared per-tick claim.
solid answer
~40 sA cron entry is local: every node that carries it fires at the scheduled minute with no knowledge of the others, so a five-node fleet runs the billing job five times a night. That means duplicate charges, duplicate invoices and five times the database load at the same instant. The naive fix, keeping the entry on one designated node, flips the failure: when that node is replaced or down, the job silently runs **zero** times. The real fix is to make the *decision to fire* cluster-wide. Either elect one leader that alone evaluates schedules, or have every node attempt a shared, atomic claim keyed by `(job, scheduled_tick)` so exactly one claim wins. Coordination can still leak a duplicate at failover, so the job should also be idempotent per tick.
go deeper
Recall that each node's scheduler fires on its own, so N nodes means N runs, and that pinning the job to one node trades that for zero runs when the node dies.
Explain the two cluster-wide mechanisms, a leader that alone runs schedules and a per-tick claim with a unique key, and why each prevents duplicate fires in the normal case.
Show that coordination still leaks duplicates at failover and lock expiry, and back it with per-tick idempotency plus an alert on any tick that has no run record.
Weigh embedding schedule coordination in every service against a shared scheduling service that gives one place to observe, audit and pause every recurring job.
## What a cron entry actually is A **cron entry** is a schedule plus a command, evaluated by a scheduler process on *one machine*. The scheduler wakes up, compares the current time with each entry's cron expression, and starts the command when they match. It knows nothing about any other machine. On a single server this is fine, and many services start that way: a nightly billing job at `0 2 * * *` (02:00 every day) and a minutely cleanup at `* * * * *` each run exactly once per tick, because there is exactly one scheduler. An in-process scheduler inside the application behaves the same way. If the application code says "every day at 02:00, run billing", every running copy of the application says it. ## Why the job multiplies when the fleet grows Modern services run as a fleet of **identical nodes** built from the same image. If the cron entry or the in-process schedule is part of that image, every node carries it. Scale to five nodes and, at 02:00, five schedulers each decide, independently and correctly by their own logic, that it is time to bill. - **Duplicate side effects**: each customer can be charged five times and receive five invoices. - **Racing writes**: five copies update the same rows, producing lost updates or constraint violations that look like random failures. - **Load multiplication**: a job sized for one runner hits the database with five times the expected load at the same instant. - **Variable behaviour**: with autoscaling, the number of runs equals whatever the node count is at 02:00, so the bug may appear only on busy nights. The root cause is simple to state: the *decision to fire* is made locally, while the *effect* is global. ## The opposite failure: zero runs A common first fix is to remove the entry from all nodes but one, or to guard it with a rule such as "only run if I am node-0". That turns "every node runs it" into "one node is special", and special nodes fail: 1. The designated node is terminated by a deploy, a scale-in or a hardware fault. 2. Replacement nodes come up from the generic image without the special setting, or the name no longer matches the rule. 3. The tick passes and **no node fires**. Nothing errors, because nothing ran, and the missing invoices surface days later. A correct design must avoid both extremes: not every node, and not one hard-coded node. ## Making the fire decision cluster-wide | Approach | How one run is chosen | Main weakness | |---|---|---| | Pin to one node | Configuration | Silent zero runs when that node dies | | Elected leader | Only the current leader evaluates schedules | Ticks can be missed or doubled around a leadership change | | Lock per tick | Every node tries an atomic claim on (job, tick); one wins | Needs a shared store and a claim that expires safely | | Dedicated scheduler service | A separate, replicated component owns all schedules | One more system to run, but one place to observe and pause jobs | The two options that live inside the application are **leader election**, where the fleet agrees on one node that alone runs the scheduler loop, and a **per-tick lock**, where every node wakes at the tick and races to insert or acquire a record keyed by the job name and the scheduled time. A unique key on that record means only one attempt succeeds. Both move the fire decision into a shared, consistent place, and both survive the loss of any single node, which pinning does not. ## Why coordination alone is not enough Coordination makes duplicates rare but cannot rule them out in every failure. A claim can expire while its holder is paused, and a leader can keep working briefly after it has lost leadership. Treat execution as **at-least-once** and make the job **idempotent per tick**: - key each unit of work by the scheduled period, for example a charge keyed by customer plus billing day, so a second run finds the work already done; - record a run row per (job, tick) so anyone can see which ticks ran and which did not; - alert when an expected tick has no run row, which is the only quick way a zero-run failure becomes visible. ## What an interviewer listens for A strong answer names **both failure directions**: N runs when every node fires, zero runs when one node was special and disappeared. It explains that the fire decision has to be shared across the fleet, names leader election and per-tick claims as the two usual mechanisms, and adds idempotency and missing-run alerting as the backstop, rather than claiming that a lock alone makes the job run exactly once.
- How would you notice that a cluster-wide cron job silently ran zero times?Absence produces no error, so you have to monitor for it. Record a run row per job and scheduled tick and alert when a tick older than its grace period has no row, or have each successful run emit a heartbeat and alert when it stops arriving, a dead-man's switch. A per-job 'last successful run' view makes gaps visible to on-call before the business notices.
- Does a per-tick lock make the nightly billing job run exactly once?No. It makes duplicate fires rare, but a claim can expire while its holder is paused and another node may then run the same tick, and a run that fails halfway may be retried. Treat execution as at-least-once and make the side effects idempotent, for example by keying each charge on customer and billing day so a repeat is a no-op.
It is like five flatmates who each set a phone reminder to pay the shared rent: all five pay. Hand the reminder to one flatmate and, the month that person is away, nobody pays.
saying these in an interview costs you the question
- Cron fires once per cluster, so adding nodes changes nothing.
- Just keep the cron entry on one designated node and move on.
- A distributed lock guarantees the job runs exactly once.
- Duplicate runs are harmless as long as the job is fast.
- No errors in the logs means the nightly job must have run.