skip to content

A Laravel task using withoutOverlapping() stopped running after its scheduler container was force-killed mid-run; why, and how do you recover and prevent it?

level: seniorimportance: should knowfreq 34%

answer

  1. SIGKILL cannot be trapped
  2. the scheduler process died, not just the child
  3. lock lives until its TTL
  4. schedule:clear-cache clears every mutex
  5. shorter expiry, heartbeat monitoring

basics

~20 s

The schedule:run process died before it could delete the task's overlap lock, so every later tick finds the lock and skips the task until its TTL, 24 hours by default, expires. Clear it with schedule:clear-cache, then set a shorter expiry and monitor.

solid answer

~50 s

`withoutOverlapping()` stores a cache lock with a TTL of `expiresAt` minutes — **1440** by default. The `schedule:run` process deletes it in the event's finish step, or from a SIGTERM/SIGINT/SIGQUIT handler for foreground tasks when `pcntl` is loaded. A force-kill (SIGKILL after the grace period, a host crash) ends `schedule:run` itself, so none of that runs and the lock survives in the shared cache. Each tick the task's skip filter sees it and dispatches `ScheduledTaskSkipped` instead of running — so nothing errors. If only the child command had been killed, `schedule:run` would have seen the non-zero exit code and released the lock normally. To recover, run `php artisan schedule:clear-cache`, which deletes every existing overlap mutex, including those of tasks really running now, so check first. To prevent it: an expiry just above the worst run time, graceful stop periods with `pcntl`, and a heartbeat such as `pingOnSuccess()` that alerts on silence.

code

php · 15 lines
php
<?php

use Illuminate\Console\Events\ScheduledTaskSkipped;
use Illuminate\Support\Facades\Event;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Schedule;

Schedule::command('reports:generate')
    ->everyFiveMinutes()
    ->withoutOverlapping(20)                      // a crash costs 20 min, not 24 h
    ->pingOnSuccess(config('services.monitor.reports_url'));

Event::listen(function (ScheduledTaskSkipped $event) {
    Log::warning('Scheduled task skipped', ['task' => $event->task->getSummaryForDisplay()]);
});

go deeper

for a junior

Remember that a crashed scheduler can leave a task's overlap lock behind, and that schedule:clear-cache removes such locks.

for a middle

Explain that the lock is released by schedule:run's finish step or signal handler, why SIGKILL bypasses both, and how the skip filter then silently skips the task.

for a senior

Tell a dead child from a dead scheduler, clear locks safely, size the expiry above the worst run, and add heartbeat monitoring that detects silence.

for a principal

Treat missed-run detection as part of each critical task's contract: an expected cadence, an alert on absence, and graceful-stop budgets in the platform.

## What the symptom looks like `reports:generate` runs every five minutes with `withoutOverlapping()`. During a deploy the orchestrator stops the scheduler container, waits out its grace period and force-kills it while a report is half built. The new container starts, cron fires `schedule:run` every minute — and the report never appears again. No exception, `schedule:run` exits normally, and `schedule:list` still shows the task with a next due time. This is the classic **stale overlap lock**. ## Who releases the lock `withoutOverlapping()` creates a mutex in the scheduler's cache store when the run starts. The code that deletes it lives in the **`schedule:run` process**, in three places: 1. The event's **finish** step, after the after-hooks, whatever the exit code. 2. The **start** step's error path, if starting the run throws. 3. A **signal handler** for SIGTERM, SIGINT and SIGQUIT — installed only for foreground tasks, only when the `pcntl` extension is loaded, and only if the second argument, `releaseOnTerminationSignals`, is left `true`. Which process dies decides the outcome: | What was killed | Lock released? | | --- | --- | | Only the child `php artisan reports:generate` process (for example, the OOM killer picked it) | Yes — `schedule:run` sees the non-zero exit code and runs the finish step | | `schedule:run` itself, with SIGTERM, and `pcntl` loaded | Yes — the signal handler deletes the lock, then exits | | `schedule:run` itself, with SIGTERM, but no `pcntl` | No | | `schedule:run` with SIGKILL, or the host or container dies | No | | A `Schedule::call()` closure hits a fatal error such as memory exhaustion | No — it dies inside `schedule:run` | SIGKILL is never delivered to user-space handlers, so PHP stops instantly. The cache entry — in Redis, the database or wherever the scheduler's store lives — keeps its full TTL: `expiresAt * 60` seconds, which is 24 hours for the default 1440 minutes. From then on, each tick the task's skip filter asks whether the mutex exists, gets `true`, and the scheduler dispatches `ScheduledTaskSkipped` instead of running it. Silence is the expected behaviour, not a bug. ## Recovering | Option | What it does | Caveat | | --- | --- | --- | | `php artisan schedule:clear-cache` | For every scheduled task whose overlap mutex exists, deletes it | Also frees the locks of tasks that are **really running**, which can start a genuine overlap | | Wait for the TTL | The cache expires the entry | Up to 24 hours of missed runs with the default | | Delete the single cache key | Removes only this task's lock | You must know the store and the mutex name (`framework/schedule-` plus a hash) | Before running `schedule:clear-cache`, confirm on every scheduler host that no protected task is running. The task then starts again at its next due minute. ## Preventing it - **Right-size the expiry.** `->withoutOverlapping(20)` for a job that normally takes seven minutes caps the damage of a crash at twenty minutes. The expiry must stay above the worst legitimate run, or a slow but healthy run loses its lock and a second copy starts. - **Let stops be graceful.** Install `pcntl` on scheduler hosts and give containers a stop grace period long enough for SIGTERM to be handled. - **Keep heavy work out of the scheduler process.** A closure that can exhaust memory takes `schedule:run` down with it; a command or queued job isolates the failure. - **Monitor absence, not just failure.** An `onFailure()` hook never fires for a run that never started. Use a heartbeat — `->pingOnSuccess($url)` to an external monitor that alerts when pings stop — or alert when `ScheduledTaskSkipped` repeats for the same task. ## Why not just turn the lock off? Setting `releaseOnTerminationSignals` to `false` makes things worse: the lock would then also survive ordinary SIGTERMs. Dropping `withoutOverlapping()` trades a missed-run failure for a concurrent-run failure. The durable fix is a short, deliberate expiry plus monitoring that notices silence.

  • If the OOM killer terminates only the child php artisan process of a Laravel scheduled command, is the overlap lock left behind?
    No. The lock is managed by the parent `schedule:run` process, which is still alive. It sees the child exit with a non-zero code, runs the event's finish step, fires the failure hooks and deletes the lock. The stale-lock case needs `schedule:run` itself to die, or an in-process closure to crash it.
  • What is the risk of setting withoutOverlapping(1) on a Laravel task that can run for ten minutes?
    The lock expires while the run is still healthy. The next tick finds no lock, takes a fresh one and starts a second copy — the overlap the option exists to prevent. The expiry has to exceed the longest legitimate run, with margin; only then is it short enough to make crashes cheap.
  • Why does an onFailure() hook not alert you to a Laravel task blocked by a stale overlap lock?
    Hooks run only when a run happens. A task blocked by its lock is skipped before it starts, so no after, success or failure hooks fire. Detecting it needs something that notices absence — an external heartbeat fed by `pingOnSuccess()` or `thenPing()`, or a listener on `ScheduledTaskSkipped`.

saying these in an interview costs you the question

  • Laravel releases the overlap lock even when schedule:run is killed with SIGKILL.
  • Any OOM kill of a scheduled command's child process leaves its lock behind.
  • A task blocked by a stale lock throws an exception every minute.
  • schedule:clear-cache only removes locks belonging to crashed tasks.
  • Setting releaseOnTerminationSignals to false prevents stale locks.
  • onFailure() hooks will alert you when a task stops running entirely.