With Hangfire, how does [AutomaticRetry] behave by default, and how do you stop a flaky payout job from retrying for days on errors that will never succeed?
answer
- a global filter, 10 attempts
- delay grows as (attempt - 1)^4 seconds
- OnAttemptsExceeded: Fail or Delete
- OnlyOn and ExceptOn filter exceptions
- the method attribute overrides the global one
basics
~20 sAutomaticRetryAttribute is global: a throwing job is retried 10 times with growing delays, about 4.5 hours in all, then left Failed. A method-level [AutomaticRetry] with fewer Attempts or ExceptOn for permanent errors stops pointless retries.
solid answer
~50 sHangfire adds `AutomaticRetryAttribute` to `GlobalJobFilters` out of the box. When a job throws, the filter replaces the Failed candidate with a Scheduled retry: `Attempts` defaults to **10**, and the delay for attempt *n* is `(n - 1)^4 + 15 + random(0..29) * n` seconds, so ten retries span roughly 4.5 hours before the job is left **Failed** (`OnAttemptsExceeded = Fail`). Someone who raised it to `Attempts = 20` made a payout retry for about six and a half days. Fix it on the method: `[AutomaticRetry(Attempts = 3, ExceptOn = new[] { typeof(PayoutRejectedException) })]` retries transient failures and fails a rejected payout immediately; `OnlyOn` is the allow-list form; `DelaysInSeconds = new[] { 30, 300, 1800 }` replaces the curve; `OnAttemptsExceeded = AttemptsExceededAction.Delete` deletes instead of failing. A method-level attribute wins over the global one because only one instance of a non-multiple filter survives. `Attempts = 0` disables retries.
code
csharp · 15 linespublic sealed class PayoutJobs(IPayoutGateway gateway, MarketplaceDbContext db)
{
// Transient faults: 3 retries on a fixed schedule.
// A rejected payout fails at once and lands in Failed for a human.
[AutomaticRetry(
Attempts = 3,
DelaysInSeconds = new[] { 60, 600, 3600 },
ExceptOn = new[] { typeof(PayoutRejectedException) })]
public async Task PaySellerAsync(long payoutId, CancellationToken ct)
{
var payout = await db.Payouts.SingleAsync(p => p.Id == payoutId, ct);
if (payout.Status == PayoutStatus.Paid) return; // safe to run again
await gateway.ExecuteAsync(payout, ct);
}
}go deeper
Remember that Hangfire retries a failing job automatically, 10 times by default, before it ends in Failed.
Explain the global filter, the growing delay curve, OnAttemptsExceeded, and how OnlyOn, ExceptOn and DelaysInSeconds tune it per job.
Classify failures as transient or permanent per job, stop retries on permanent ones, and pair any retry policy with a job that is safe to repeat.
Set the organisation's retry policy for background work: budgets per job type, alerting when retries begin rather than end, and who owns Failed jobs.
## Retries are a job filter Hangfire implements retries as an ordinary **job filter**, `AutomaticRetryAttribute`, which implements `IElectStateFilter`. It is registered in `GlobalJobFilters.Filters` by default, so every job gets it without any attribute. When a job throws, the worker proposes a `FailedState`; during state election the filter decides whether to replace it: 1. If the exception does not match `OnlyOn` (when set) or matches `ExceptOn`, the filter steps aside and the job is **Failed** at once. 2. Otherwise it reads the job's `RetryCount` parameter, adds one and, while that is within `Attempts`, elects a **Scheduled** state with a delay and the reason "Retry attempt n of N: message". 3. Past `Attempts`, it either leaves the job **Failed** (default) or moves it to **Deleted** when `OnAttemptsExceeded = AttemptsExceededAction.Delete`. ## The defaults, in numbers | Property | Default | |---|---| | `Attempts` | 10 | | delay for attempt n | `(n - 1)^4 + 15 + random(0..29) * n` seconds | | `OnAttemptsExceeded` | `AttemptsExceededAction.Fail` | | `LogEvents` | `true` (warning per retry, error at the end) | | `OnlyOn` / `ExceptOn` | not set: every exception is retried | The curve starts at 15 to 44 seconds and reaches roughly 1.8 hours for the tenth attempt; ten retries add up to about 4.5 hours. With 20 attempts the sum is about 157 hours, close to a week, which is how a job ends up retrying for days. ## Why the default hurts a payout job Retrying blindly is right for transient faults (a timeout, a deadlock, a 503) and wrong for permanent ones (an invalid IBAN, a closed seller account, a rejected payout). For a marketplace payout, ten retries of a rejected request means ten calls to the payment provider, hours of noise, and an operator who only learns of the problem at the end. ## Tuning it per job - **`ExceptOn`**: exception types that must not be retried, such as `PayoutRejectedException` or a validation error. Matching includes derived types. - **`OnlyOn`**: the opposite, retry only these, for example `HttpRequestException` and `TimeoutException`. - **`Attempts`**: fewer for jobs with a user waiting, `0` to disable retries completely. - **`DelaysInSeconds`**: an explicit schedule such as `[30, 300, 1800]`; attempts beyond its length reuse the last value. - **`OnAttemptsExceeded = AttemptsExceededAction.Delete`**: for jobs whose failure is acceptable and should not clutter the Failed list. Use it sparingly; Deleted jobs expire after a day and nobody looks at them. ## Where the attribute goes and which one wins The attribute can sit on the job **method** or on the job's **type**. Because `AutomaticRetryAttribute` does not allow multiple instances, Hangfire keeps one: filters are ordered by `Order` and then scope (global, then type, then method), and duplicates are removed from the least specific side. A method attribute therefore overrides the global default. To change the default for all jobs, register a configured instance with `config.UseFilter(new AutomaticRetryAttribute { Attempts = 3 })` in `AddHangfire`. ## Retries need idempotent jobs Every retry runs the whole method again. A payout that called the provider and then failed while saving the result will call it again unless the job checks first. Tuning retries limits the damage; idempotent job design prevents it.
- How do you change the retry default for every job instead of decorating each method?Register a configured instance globally in `AddHangfire`: `config.UseFilter(new AutomaticRetryAttribute { Attempts = 3 })`. Only one `AutomaticRetryAttribute` survives per job, so it supersedes the built-in default, and a method-level attribute still overrides it for a specific job.
- Does a retried job keep its job id?Yes. A retry is the same background job moving from Failed-candidate to Scheduled and back to Enqueued; its id, arguments and parameters (including RetryCount) are unchanged. That stable id is useful as an idempotency key toward external systems.
- Hangfire does not retry failed jobs unless you add [AutomaticRetry].
- After its retries a job is deleted by default.
- The default 10 retries happen within a few minutes.
- ExceptOn exceptions are retried once and then failed.
- Setting Attempts = 0 makes Hangfire retry forever.