skip to content

With Solid Queue, what do the workers and dispatchers entries in config/queue.yml do, and what do threads, processes and polling_interval control?

level: middleimportance: must knowfreq 48%

answer

  1. supervisor, workers, dispatchers, scheduler
  2. dispatchers move due scheduled jobs
  3. threads per process, processes forked
  4. queue list order is strict
  5. pool needs threads plus 2

basics

~20 s

In Solid Queue, workers poll their queues for ready jobs and run them in a thread pool; dispatchers move due scheduled jobs to ready and maintain concurrency limits. threads sizes each pool, processes forks worker copies, polling_interval spaces the polls.

solid answer

~40 s

The Solid Queue supervisor reads `config/queue.yml` for the current environment. Each **worker** entry polls the ready executions of its `queues` (`"*"` by default) with `FOR UPDATE SKIP LOCKED` and runs claimed jobs in a pool of `threads` (default 3); `processes` (default 1) forks that many identical worker processes, and is ignored in `async` mode. A list such as `[digests, maintenance]` is strict order: `maintenance` is polled only when `digests` is empty. **Dispatchers** move scheduled jobs that are due from `solid_queue_scheduled_executions` to the ready table in batches of `batch_size` (500) and run concurrency maintenance. `polling_interval` is the wait between polls: 0.1 s for workers and 1 s for dispatchers by default, while the generated file sets 1 s for both and `processes` from `JOB_CONCURRENCY`. Each worker process needs about `threads + 2` database connections.

code

bash · 4 lines
bash
bin/jobs check                 # validate queue.yml and recurring.yml, exit non-zero on errors
bin/jobs -c config/queue.yml   # start the supervisor with an explicit config file
bin/jobs --mode async          # run workers and dispatchers as threads; processes is ignored
JOB_CONCURRENCY=3 bin/jobs     # the generated queue.yml reads processes from this

go deeper

for a junior

Know that queue.yml defines workers and dispatchers, and that threads and processes set how many jobs run at once.

for a middle

Explain each actor's table, the defaults, ordered queue lists, and what the generated file changes compared with Solid Queue's defaults.

for a senior

Size threads, processes and the queue database's connections together, and choose processes for CPU-bound queues and threads for I/O-bound ones.

for a principal

Decide how queues map to worker pools so that urgent work cannot be starved by bulk work as load grows.

## The four kinds of process Solid Queue splits the work of a job backend across separate actors, all started and watched by one **supervisor**: - **Workers** pick jobs that are ready to run from `solid_queue_ready_executions` and perform them. - **Dispatchers** find **scheduled** jobs that are due (enqueued with `set(wait:)` or `wait_until:`) and move them from `solid_queue_scheduled_executions` to the ready table. They also run **concurrency maintenance**, releasing jobs blocked by concurrency limits. - The **scheduler** enqueues the recurring tasks from `config/recurring.yml`; it is only started when such tasks exist. - The **supervisor**, started by `bin/jobs` or the Puma plugin, forks the others (in the default `fork` mode), tracks their heartbeats and replaces any that die. `config/queue.yml` configures the first two, keyed by environment. ## What the installer generates ```yaml default: &default dispatchers: - polling_interval: 1 batch_size: 500 workers: - queues: "*" threads: 3 processes: <%= ENV.fetch("JOB_CONCURRENCY", 1) %> polling_interval: 1 ``` One dispatcher and one worker process with three threads serving every queue. With no configuration at all, Solid Queue would start the same shape with its own defaults. ## The options | Option | Applies to | Default | Meaning | |---|---|---|---| | `queues` | workers | `"*"` | which queues to poll, in order | | `threads` | workers | 3 | size of the job thread pool in each process | | `processes` | workers | 1 | how many identical worker processes to fork | | `polling_interval` | both | 0.1 s workers, 1 s dispatchers | seconds to wait between polls | | `batch_size` | dispatchers | 500 | due jobs moved per dispatch | | `concurrency_maintenance_interval` | dispatchers | 600 s | how often blocked jobs are checked | Workers can use `fibers: N` instead of `threads` for I/O-bound jobs, never both, and that needs fiber-scoped execution state in Rails. ## Queue lists are ordered For a newsletter platform that must get digests out before anything else: ```yaml production: dispatchers: - polling_interval: 1 batch_size: 500 workers: - queues: [ digests, maintenance ] threads: 5 processes: 2 - queues: default threads: 3 ``` 1. The first worker definition starts **two processes with five threads each**, ten job threads in all. 2. It takes a `maintenance` job **only when no ready `digests` job remains**. The list is strict priority, not round robin, and queue order beats Active Job priorities. 3. `*` means every queue; a prefix like `mail*` is allowed, but `*` cannot appear in the middle or at the start of a name. Exact names poll fastest, because prefixes and paused queues need an extra `DISTINCT` query. ## Sizing against the database Each worker process uses connections for its job threads, its polling and its heartbeat, so it needs roughly **`threads + 2`** connections to the queue database. Solid Queue logs a warning at boot, and `bin/jobs check` prints the same one, when the pool is smaller. Adding `processes` multiplies the number of connections the database must accept, not the pool each process needs. ## Choosing numbers for a real workload For the newsletter platform the three kinds of work behave differently: - **Sending digests** mostly waits on the email provider's API, so it is I/O-bound: more `threads` per process helps, up to what the provider and the connection pool allow. - **Rendering digests** from templates burns CPU: on CRuby, threads in one process do not run Ruby code in parallel, so extra `processes` are what add throughput. - **Maintenance** such as purging bounced subscribers should not compete with either: a small worker with one or two threads on its own queue keeps it from crowding out digests. Every extra process costs memory and `threads + 2` connections, so the totals have to fit the server and the database. ## Fork versus async - In the default **`fork`** mode, every worker and dispatcher is its own OS process: isolation and true parallelism on CRuby, more memory. - With `bin/jobs --mode async` or `SOLID_QUEUE_SUPERVISOR_MODE=async`, they run as threads inside the supervisor and `processes` is **ignored**. The project recommends `fork` unless you have a strong reason.

  • Why would you raise processes rather than threads for a CPU-heavy queue?
    On CRuby, threads in one process share a global VM lock, so they overlap I/O waits but do not run Ruby code in parallel. Forked worker processes each get their own interpreter and can use separate cores. Threads suit jobs that mostly wait on SMTP or HTTP; processes suit jobs that render or compute, at the cost of memory and connections.
  • What happens to jobs enqueued with set(wait: 1.hour) if you run workers but no dispatcher?
    They stay in the scheduled executions table forever. Workers only poll ready executions; moving a due scheduled job to ready is the dispatcher's job. A configuration that lists only workers starts no dispatcher, so delayed jobs never become runnable, and blocked jobs lose the failsafe release that concurrency maintenance provides.

  • processes: 2 with threads: 5 means five threads split across two processes
  • A worker with queues [a, b] alternates between the two queues
  • Workers move scheduled jobs to ready when they are due
  • polling_interval is how long a job may run before it is killed
  • processes still forks copies when the supervisor runs in async mode