When several Hangfire servers share one PostgreSQL database, what stops two of them running the same job or recurring occurrence, and how do Hangfire's distributed locks work?
answer
- atomic fetch with SKIP LOCKED
- schedulers guarded by named locks
- SQL Server: sp_getapplock per session
- PostgreSQL: rows in the lock table
- DistributedLockTimeoutException on contention
basics
~20 sEach queue row is claimed atomically, so only one worker fetches a job, and every server's schedulers take a storage-level distributed lock before promoting delayed or recurring jobs. The lock is a SQL Server application lock, a PostgreSQL lock-table row or a Redis key.
solid answer
~50 sTwo mechanisms cooperate. **Fetching** is atomic in the storage: Hangfire.PostgreSql claims a queue row with `UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1)`, Hangfire.SqlServer with `UPDATE TOP (1) ... WITH (READPAST, UPDLOCK)`, so one row goes to one worker. **Scheduling** runs on every server, but the delayed-job scheduler and the recurring-job scheduler each take a **distributed lock** (`locks:schedulepoller`, `recurring-jobs:lock`) and recurring definitions are locked per id, so one occurrence becomes one job. The lock is storage-specific: SQL Server uses session-owned `sp_getapplock`, released when the connection drops; Hangfire.PostgreSql inserts a row into its `lock` table and treats rows older than `DistributedLockTimeout` (10 minutes) as stale; the community Redis storage uses an expiring key it keeps extending. A waiter that cannot acquire in time gets `DistributedLockTimeoutException`. None of this makes a job exactly-once; invisibility timeouts can still repeat it.
code
sql · 13 lines-- Shape of Hangfire.PostgreSql's fetch (schema "hangfire", InvisibilityTimeout 30 min)
UPDATE "hangfire"."jobqueue"
SET "fetchedat" = NOW()
WHERE "id" = (
SELECT "id"
FROM "hangfire"."jobqueue"
WHERE "queue" = ANY (@Queues)
AND ("fetchedat" IS NULL OR "fetchedat" < NOW() + INTERVAL '-1800 SECONDS')
ORDER BY "fetchedat" NULLS FIRST, "queue", "jobid"
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING "id", "jobid", "queue", "fetchedat";go deeper
Remember that several Hangfire servers can share one database safely; the storage hands each job to one worker.
Explain the atomic fetch, the named locks around the delayed and recurring schedulers, and DistributedLockTimeoutException.
Compare lock behaviour per storage when a holder dies, keep locked work short relative to DistributedLockTimeout, and use lightweight servers in big fleets.
Judge how far a shared database can carry coordination for the fleet, and when lock contention or polling load argues for a different storage.
## The problem A scaled-out marketplace runs, say, four instances, each with `AddHangfireServer`, all pointed at one PostgreSQL database. Every server has workers polling the same queues and its own copy of the schedulers. Without coordination the same payout could be fetched four times, a delayed job promoted four times and a nightly settlement fired four times. Hangfire relies on the database for coordination; servers never talk to each other. ## Atomic fetching A job becomes runnable as a row in the storage's **job queue** table. Claiming it is a single atomic statement: - **Hangfire.PostgreSql**: an `UPDATE ... SET fetchedat = NOW()` whose target is chosen by a sub-select using `FOR UPDATE SKIP LOCKED LIMIT 1`. Concurrent workers skip rows another transaction has locked instead of waiting on them. - **Hangfire.SqlServer**: `UPDATE TOP (1)` with `READPAST, UPDLOCK, ROWLOCK` hints, setting `FetchedAt`, with the same effect. A claimed row stays invisible to others until it is acknowledged or its invisibility timeout passes, which is where duplicates can still come from. ## Distributed locks around the schedulers Every full server runs the **delayed job scheduler** and the **recurring job scheduler**. Each wraps its pass in a named **distributed lock** obtained from the storage connection: | Process | Lock resource | |---|---| | Delayed job scheduler | `locks:schedulepoller` | | Recurring job scheduler | `recurring-jobs:lock` | | `AddOrUpdate`, `TriggerJob`, `RemoveIfExists` | a lock per recurring job id | Only one server performs a pass at a time; the others find the lock taken and try again on their next tick. A server created with `IsLightweightServer = true` runs workers only, without schedulers or storage maintenance, which reduces lock traffic in large fleets. ## How each storage implements a lock | Storage | Mechanism | If the holder dies | |---|---|---| | Hangfire.SqlServer | `sp_getapplock`, exclusive, owned by the session | released when its connection closes | | Hangfire.PostgreSql | a row in the `lock` table keyed by resource | treated as stale after `DistributedLockTimeout` (10 min) | | Hangfire.Redis.StackExchange | an expiring key extended while held | expires when no longer extended | The paid Hangfire.Pro.Redis storage has its own implementation, which is not open source. A caller that cannot get a lock within its timeout receives `DistributedLockTimeoutException`. The same mechanism backs `[DisableConcurrentExecution]` on job methods. ## What the locks do not promise 1. They coordinate Hangfire's own bookkeeping; they do not stop your job body from running twice after a crash or a long job outliving its invisibility timeout. 2. PostgreSQL's stale-lock rule means a lock can be taken over after 10 minutes even if the original holder is merely slow, so a lock-protected section should finish well within `DistributedLockTimeout`. 3. Locks add database round trips; dozens of servers with full schedulers contend on the same rows, which is a reason to keep scheduler-running servers few and the rest lightweight. ## Takeaways - Scaling out by adding servers is safe: the database arbitrates. - Duplicates come from timeouts, not from concurrent servers. - Know which lock implementation your storage uses before relying on it for anything long.
- Four instances all register the same recurring job at startup and all run the recurring scheduler; why does the nightly settlement still run once per night?Registration is an upsert under a per-id lock, so there is one definition. Each scheduler pass runs under `recurring-jobs:lock`, and the definition's next execution time is advanced in the same pass, so only one server creates the job for that occurrence.
- When would you use IsLightweightServer?In a large fleet where only a few servers need to run schedulers and storage maintenance. Lightweight servers run workers only, cutting lock contention and polling; at least one full server must remain, or delayed and recurring jobs stop being promoted. Hangfire.PostgreSql's sliding invisibility timeout also does not work on lightweight servers.
- Hangfire servers elect a leader over the network to divide jobs.
- Only one server may run against a database at a time.
- Distributed locks guarantee that a job body never runs twice.
- A PostgreSQL storage lock lives until it is released, however long that takes.
- Every server fires each recurring job, so it runs once per instance.