Background Jobs & Scheduling
Getting work off the request path: task queues, background workers and scheduled jobs in .NET and Python. Asked because retries and at-least-once delivery are where it silently fails.
on this pageshowhide
explore
- Hangfire18 questions
- Job Types & Recurring Runs6 questions
- Storage & Servers6 questions
- Dashboard & Retries6 questions
- Quartzempty
- Coravelempty
- Celery43 questions
- Defining & Calling Tasks6 questions
- Task Canvas6 questions
- Execution Pools & Prefetch5 questions
- Brokers and Result Backends6 questions
- Routing and Queues5 questions
- Retries & Acknowledgement5 questions
- Beat Scheduling5 questions
- Monitoring & Shutdown5 questions
questions
61 · 2 sectionsWith Hangfire, what is the difference between BackgroundJob.Enqueue, BackgroundJob.Schedule and RecurringJob.AddOrUpdate, and when do you use each?
basics
~20 sBackgroundJob.Enqueue creates a fire-and-forget job that runs as soon as a worker is free; BackgroundJob.Schedule creates a job that runs after a delay or at a set time; RecurringJob.AddOrUpdate registers a cron schedule that creates a new job at every occurrence.
In an ASP.NET Core 10 app, how do you wire Hangfire with AddHangfire, a PostgreSQL job storage and AddHangfireServer, and what does each call do?
basics
~20 sAddHangfire configures Hangfire globally, sets the job storage (UsePostgreSqlStorage for PostgreSQL) and registers the job clients. AddHangfireServer adds a hosted service that runs a processing server: workers that execute jobs plus the schedulers, all reading the same storage.
With Hangfire, why should a job method take an order id rather than the Order entity, and what happens to its arguments at enqueue time?
basics
~20 sHangfire evaluates a job's arguments and serializes them to JSON in storage when the job is created, so an entity becomes a stale, bulky snapshot. Passing the id lets the job load current data when it actually runs.
With Hangfire, how does [AutomaticRetry] behave by default, and how do you stop a flaky payout job from retrying for days on errors that will never succeed?
basics
~20 sAutomaticRetryAttribute is global: a throwing job is retried 10 times with growing delays, about 4.5 hours in all, then left Failed. A method-level [AutomaticRetry] with fewer Attempts or ExceptOn for permanent errors stops pointless retries.
With Hangfire on ASP.NET Core, why does the dashboard work on a developer machine but return 401 once deployed, and how do you secure it properly?
basics
~10 sDashboardOptions.Authorization defaults to LocalRequestsOnlyAuthorizationFilter, which admits only loopback or same-IP requests, so a deployed dashboard answers 401. Replace it with your own IDashboardAuthorizationFilter, or map it with MapHangfireDashboardWithAuthorizationPolicy and an ASP.NET Core policy.
In Celery, what does the celery beat process do, and how do you declare a periodic task in the beat_schedule setting?
basics
~20 scelery beat is a scheduler process that sends a task message whenever a beat_schedule entry comes due; workers run it. Each entry names a registered task, a schedule (seconds, timedelta, crontab or solar) and optional args, kwargs and options.
In Celery, what is the difference between the broker and the result backend, and does every app need a result backend?
basics
~20 sCelery's broker carries task messages from the caller to the workers; the result backend stores each task's state and return value. The broker is required; the result backend is optional and unset by default, so fire-and-forget tasks need none.
In Celery, how does `self.retry()` re-run a failing task, and what do its `countdown`, `max_retries` and `exc` arguments control?
basics
~20 sA bound Celery task calls raise self.retry(exc=exc, countdown=N) in its except block: Celery publishes a new message for the same task id, delayed N seconds (180 by default), up to max_retries (3), then re-raises exc as the failure.
In Celery, how does calling a task with delay() differ from apply_async(), and what does either call return to the caller?
basics
~20 sdelay(*args, **kwargs) is a shortcut for apply_async(args, kwargs) with no execution options. Both serialize the arguments, publish one message to the broker and return an AsyncResult with the task id at once; neither waits for a worker.
In Celery, how do chain, group and chord differ, and which fits resizing thumbnails in parallel and then running one step after all finish?
basics
~20 sIn Celery, a chain runs tasks in sequence and passes each result on; a group runs tasks in parallel; a chord is a group plus one body task that receives the list of all header results. Parallel thumbnails, then publish, is a chord.