skip to content

Dashboard & Retries

Operating Hangfire: the dashboard and its authorization filters, job states, [AutomaticRetry] with 10 default attempts, and job filters. Interviewers ask why a deployed dashboard returns 401.

on this pageshow

explore

questions

6

With Hangfire, how does [AutomaticRetry] behave by default, and how do you stop a flaky payout job from retrying for days on errors that will never succeed?

level: middleimportance: must knowfreq 50%

answer

  1. a global filter, 10 attempts
  2. delay grows as (attempt - 1)^4 seconds
  3. OnAttemptsExceeded: Fail or Delete
  4. OnlyOn and ExceptOn filter exceptions
  5. the method attribute overrides the global one

basics

~20 s

AutomaticRetryAttribute is global: a throwing job is retried 10 times with growing delays, about 4.5 hours in all, then left Failed. A method-level [AutomaticRetry] with fewer Attempts or ExceptOn for permanent errors stops pointless retries.

solid answer

~50 s

Hangfire adds `AutomaticRetryAttribute` to `GlobalJobFilters` out of the box. When a job throws, the filter replaces the Failed candidate with a Scheduled retry: `Attempts` defaults to **10**, and the delay for attempt *n* is `(n - 1)^4 + 15 + random(0..29) * n` seconds, so ten retries span roughly 4.5 hours before the job is left **Failed** (`OnAttemptsExceeded = Fail`). Someone who raised it to `Attempts = 20` made a payout retry for about six and a half days. Fix it on the method: `[AutomaticRetry(Attempts = 3, ExceptOn = new[] { typeof(PayoutRejectedException) })]` retries transient failures and fails a rejected payout immediately; `OnlyOn` is the allow-list form; `DelaysInSeconds = new[] { 30, 300, 1800 }` replaces the curve; `OnAttemptsExceeded = AttemptsExceededAction.Delete` deletes instead of failing. A method-level attribute wins over the global one because only one instance of a non-multiple filter survives. `Attempts = 0` disables retries.

code

csharp · 15 lines
csharp
public sealed class PayoutJobs(IPayoutGateway gateway, MarketplaceDbContext db)
{
    // Transient faults: 3 retries on a fixed schedule.
    // A rejected payout fails at once and lands in Failed for a human.
    [AutomaticRetry(
        Attempts = 3,
        DelaysInSeconds = new[] { 60, 600, 3600 },
        ExceptOn = new[] { typeof(PayoutRejectedException) })]
    public async Task PaySellerAsync(long payoutId, CancellationToken ct)
    {
        var payout = await db.Payouts.SingleAsync(p => p.Id == payoutId, ct);
        if (payout.Status == PayoutStatus.Paid) return;   // safe to run again
        await gateway.ExecuteAsync(payout, ct);
    }
}

go deeper

for a junior

Remember that Hangfire retries a failing job automatically, 10 times by default, before it ends in Failed.

for a middle

Explain the global filter, the growing delay curve, OnAttemptsExceeded, and how OnlyOn, ExceptOn and DelaysInSeconds tune it per job.

for a senior

Classify failures as transient or permanent per job, stop retries on permanent ones, and pair any retry policy with a job that is safe to repeat.

for a principal

Set the organisation's retry policy for background work: budgets per job type, alerting when retries begin rather than end, and who owns Failed jobs.

## Retries are a job filter Hangfire implements retries as an ordinary **job filter**, `AutomaticRetryAttribute`, which implements `IElectStateFilter`. It is registered in `GlobalJobFilters.Filters` by default, so every job gets it without any attribute. When a job throws, the worker proposes a `FailedState`; during state election the filter decides whether to replace it: 1. If the exception does not match `OnlyOn` (when set) or matches `ExceptOn`, the filter steps aside and the job is **Failed** at once. 2. Otherwise it reads the job's `RetryCount` parameter, adds one and, while that is within `Attempts`, elects a **Scheduled** state with a delay and the reason "Retry attempt n of N: message". 3. Past `Attempts`, it either leaves the job **Failed** (default) or moves it to **Deleted** when `OnAttemptsExceeded = AttemptsExceededAction.Delete`. ## The defaults, in numbers | Property | Default | |---|---| | `Attempts` | 10 | | delay for attempt n | `(n - 1)^4 + 15 + random(0..29) * n` seconds | | `OnAttemptsExceeded` | `AttemptsExceededAction.Fail` | | `LogEvents` | `true` (warning per retry, error at the end) | | `OnlyOn` / `ExceptOn` | not set: every exception is retried | The curve starts at 15 to 44 seconds and reaches roughly 1.8 hours for the tenth attempt; ten retries add up to about 4.5 hours. With 20 attempts the sum is about 157 hours, close to a week, which is how a job ends up retrying for days. ## Why the default hurts a payout job Retrying blindly is right for transient faults (a timeout, a deadlock, a 503) and wrong for permanent ones (an invalid IBAN, a closed seller account, a rejected payout). For a marketplace payout, ten retries of a rejected request means ten calls to the payment provider, hours of noise, and an operator who only learns of the problem at the end. ## Tuning it per job - **`ExceptOn`**: exception types that must not be retried, such as `PayoutRejectedException` or a validation error. Matching includes derived types. - **`OnlyOn`**: the opposite, retry only these, for example `HttpRequestException` and `TimeoutException`. - **`Attempts`**: fewer for jobs with a user waiting, `0` to disable retries completely. - **`DelaysInSeconds`**: an explicit schedule such as `[30, 300, 1800]`; attempts beyond its length reuse the last value. - **`OnAttemptsExceeded = AttemptsExceededAction.Delete`**: for jobs whose failure is acceptable and should not clutter the Failed list. Use it sparingly; Deleted jobs expire after a day and nobody looks at them. ## Where the attribute goes and which one wins The attribute can sit on the job **method** or on the job's **type**. Because `AutomaticRetryAttribute` does not allow multiple instances, Hangfire keeps one: filters are ordered by `Order` and then scope (global, then type, then method), and duplicates are removed from the least specific side. A method attribute therefore overrides the global default. To change the default for all jobs, register a configured instance with `config.UseFilter(new AutomaticRetryAttribute { Attempts = 3 })` in `AddHangfire`. ## Retries need idempotent jobs Every retry runs the whole method again. A payout that called the provider and then failed while saving the result will call it again unless the job checks first. Tuning retries limits the damage; idempotent job design prevents it.

  • How do you change the retry default for every job instead of decorating each method?
    Register a configured instance globally in `AddHangfire`: `config.UseFilter(new AutomaticRetryAttribute { Attempts = 3 })`. Only one `AutomaticRetryAttribute` survives per job, so it supersedes the built-in default, and a method-level attribute still overrides it for a specific job.
  • Does a retried job keep its job id?
    Yes. A retry is the same background job moving from Failed-candidate to Scheduled and back to Enqueued; its id, arguments and parameters (including RetryCount) are unchanged. That stable id is useful as an idempotency key toward external systems.

  • Hangfire does not retry failed jobs unless you add [AutomaticRetry].
  • After its retries a job is deleted by default.
  • The default 10 retries happen within a few minutes.
  • ExceptOn exceptions are retried once and then failed.
  • Setting Attempts = 0 makes Hangfire retry forever.
open as a page

With Hangfire on ASP.NET Core, why does the dashboard work on a developer machine but return 401 once deployed, and how do you secure it properly?

level: middleimportance: must knowfreq 52%

basics

~10 s

DashboardOptions.Authorization defaults to LocalRequestsOnlyAuthorizationFilter, which admits only loopback or same-IP requests, so a deployed dashboard answers 401. Replace it with your own IDashboardAuthorizationFilter, or map it with MapHangfireDashboardWithAuthorizationPolicy and an ASP.NET Core policy.

open as a page

In Hangfire, which states can a background job be in, how does it move between them, and which ones expire on their own?

level: juniorimportance: should knowfreq 48%

basics

~20 s

A job starts Enqueued, Scheduled or Awaiting, becomes Processing on a worker, and ends Succeeded, Failed or Deleted. Succeeded and Deleted are final and expire after a day; Failed stays until someone requeues or deletes it.

open as a page

With Hangfire, what does [DisableConcurrentExecution(timeoutInSeconds)] actually guarantee, and when is a dedicated queue with one worker the better tool?

level: seniorimportance: should knowfreq 30%

basics

~20 s

[DisableConcurrentExecution] holds a storage distributed lock while the job method runs, so runs sharing a lock resource never overlap. It does not skip duplicates: the second run waits on its worker and fails into retries after the timeout.

open as a page

With Hangfire, when can a seller-payout job's method run more than once, and how do you write it so a second run pays nobody twice?

level: seniorimportance: should knowfreq 38%

basics

~20 s

A Hangfire job reruns on every automatic retry, after a crash or expired invisibility timeout, and on a manual requeue. The payout job checks its recorded state first and sends the provider a key that stays stable across runs.

open as a page

With Hangfire, how do job filters such as IClientFilter, IServerFilter and IElectStateFilter work, and how would you add one that alerts when a payout job finally fails?

level: seniorimportance: nice to knowfreq 22%

basics

~10 s

Job filters are Hangfire's hooks around creation (IClientFilter), execution (IServerFilter) and state changes (IElectStateFilter, IApplyStateFilter). Apply one as a JobFilterAttribute on a method or class, or globally with config.UseFilter; a final-failure alert fits IApplyStateFilter.

open as a page