With Laravel Horizon, a burst of notification jobs leaves billing jobs waiting although one auto-balanced supervisor serves both queues; how do balance strategies explain this, and how would you fix it?

seniormust knowfreq 45%

answer

queue order is ignored under auto
time = size x average runtime
size, time or log strategy
balance false = strict listed order
separate supervisor, own min and max

basics

~20 s

Auto balancing splits a supervisor's maxProcesses by queue load, not list order, so a notifications flood takes most workers while billing keeps minProcesses. Give billing its own supervisor with a guaranteed floor, or use balance false for strict order.

solid answer

~40 s

Under balance => 'auto' Horizon keeps a worker pool per queue and every balanceCooldown seconds resizes them in proportion to load, using autoScalingStrategy: time (backlog times average runtime, the default), size (job count) or log (logarithm of the count). The order of the queue list gives no priority, so 40,000 notifications pull the pools toward that queue while billing sits at minProcesses, and a queue's pool can only grow by balanceMaxShift processes per cooldown. Fixes: move billing to its own supervisor with its own minProcesses/maxProcesses so it has a guaranteed floor, and cap the notifications supervisor; or use balance => false, where one pool works the queues strictly in listed order; simple splits a fixed count evenly. log only softens the skew.

Under `balance => 'auto'` Horizon keeps a worker pool per queue and every `balanceCooldown` seconds resizes them in proportion to load, using `autoScalingStrategy`: `time` (backlog times average runtime, the default), `size` (job count) or `log` (logarithm of the count). The order of the `queue` list gives no priority, so 40,000 notifications pull the pools toward that queue while billing sits at `minProcesses`, and a queue's pool can only grow by `balanceMaxShift` processes per cooldown. Fixes: move billing to its own supervisor with its own `minProcesses`/`maxProcesses` so it has a guaranteed floor, and cap the notifications supervisor; or use `balance => false`, where one pool works the queues strictly in listed order; `simple` splits a fixed count evenly. `log` only softens the skew.

code

php · 23 lines

php

<?php

// config/horizon.php (excerpt)
'environments' => [
    'production' => [
        'supervisor-billing' => [
            'connection' => 'redis',
            'queue' => ['billing'],
            'balance' => 'auto',
            'minProcesses' => 2,
            'maxProcesses' => 4,
            'timeout' => 120,
        ],
        'supervisor-notifications' => [
            'connection' => 'redis',
            'queue' => ['notifications'],
            'balance' => 'auto',
            'autoScalingStrategy' => 'size',
            'minProcesses' => 1,
            'maxProcesses' => 6,
        ],
    ],
],

go deeper

for a junior

Recall the three balance values, auto, simple and false, and that auto is the published default for a Horizon supervisor.

for a middle

Explain that auto sizes one pool per queue by load, that minProcesses is per queue and maxProcesses a total, and what the time strategy measures.

for a senior

Diagnose starvation from the config, split supervisors to protect a critical queue, and keep timeouts safe for scale-down.

for a principal

Decide which workloads deserve isolated worker budgets and whether throughput or strict priority matters more for each business flow.

The symptom

A production supervisor serves two queues on one Redis connection:

billing - a steady trickle of charge and invoice jobs that customers wait on;
notifications - usually quiet, but a marketing send can drop tens of thousands of jobs at once.

Both sit under one supervisor with balance => 'auto', minProcesses => 1 and maxProcesses => 10. During a send, billing latency climbs from seconds to minutes although Horizon shows ten busy workers. Nothing is broken: this is auto balancing doing exactly what it was configured to do.

What each balance value does

Every supervisor chooses one of three strategies with its balance key:

`balance`	Worker pools	Process count	Queue order
`auto` (published default)	one pool per queue	scales per queue between `minProcesses` and a shared `maxProcesses`	ignored; load decides
`simple`	one pool per queue	fixed: `maxProcesses` (alias `processes`) split evenly	ignored
`false`	one pool for all queues	scales between `minProcesses` and `maxProcesses` with the backlog	strict: each worker drains the first listed queue first

With auto, minProcesses is per queue and maxProcesses is the total across the supervisor's queues. With false, both are totals.

Why auto starves billing

Every balanceCooldown seconds (3 by default) the supervisor asks Horizon's auto-scaler for a target per queue:

It reads each queue's ready backlog from Redis.
It weighs the queues with the autoScalingStrategy:
- time (default) - backlog multiplied by the queue's average job runtime, i.e. estimated time to clear;
- size - the raw number of jobs waiting;
- log - the logarithm of the job count (log1p), so a huge queue gets a less disproportionate share.
It multiplies each queue's share by maxProcesses.
It moves each pool toward its target by at most balanceMaxShift processes (1 by default) per cycle, never below minProcesses and never above the supervisor's total.

A worked example with maxProcesses => 10 and minProcesses => 1: billing has 20 jobs averaging 2 seconds (40 seconds of work), notifications has 40,000 jobs averaging 50 ms (2,000 seconds). Under time, billing's share is about 2%, so its target rounds up to one process, and notifications climbs to nine, the most it may take while billing keeps its floor. Under size the split is the same. Under log the weights are log1p(20) (about 3) and log1p(40000) (about 10.6), so billing's share rises to about a fifth and it holds two or three processes instead of one: better, but still load-driven, not guaranteed.

Listing billing first changes nothing, because under auto the list order is not a priority.

Fixes, in order of preference

Split supervisors. Give billing its own supervisor with, say, minProcesses => 2 and maxProcesses => 4, and put notifications in a second supervisor with maxProcesses => 6. Each supervisor has its own budget, so a flood in one cannot take the other's processes.
Strict priority with balance => false. One pool works ['billing', 'notifications'] in that order: billing is always drained first and notifications wait. Use it only when starving the low-priority queue is acceptable.
Fixed split with simple. processes => 10 over two queues gives five each: predictable, but idle billing workers cannot help notifications.
Soften with log. Keeps one supervisor but still lets the bigger backlog win; a mitigation, not a guarantee.

Also raise minProcesses if a single billing worker cannot keep up with its steady rate on its own.

The scale-down caveat

When auto shrinks a pool, Horizon asks a worker to stop after its current job; if it is still running after the supervisor's timeout seconds, Horizon treats it as hanging and kills it. The docs therefore say to keep the supervisor timeout above any job-level timeout, so rebalancing never cuts a long billing job in half.

Why this is a Horizon question

Plain queue:work with --queue=billing,notifications gives strict order and a fixed process count set outside Laravel. Horizon's value is the dynamic middle ground, and the interview probe is whether you know that its default dynamic mode trades priority for throughput.

## The symptom A production supervisor serves two queues on one Redis connection: - `billing` - a steady trickle of charge and invoice jobs that customers wait on; - `notifications` - usually quiet, but a marketing send can drop tens of thousands of jobs at once. Both sit under one supervisor with `balance => 'auto'`, `minProcesses => 1` and `maxProcesses => 10`. During a send, billing latency climbs from seconds to minutes although Horizon shows ten busy workers. Nothing is broken: this is auto balancing doing exactly what it was configured to do. ## What each balance value does Every supervisor chooses one of three strategies with its `balance` key: | `balance` | Worker pools | Process count | Queue order | |---|---|---|---| | `auto` (published default) | one pool per queue | scales per queue between `minProcesses` and a shared `maxProcesses` | ignored; load decides | | `simple` | one pool per queue | fixed: `maxProcesses` (alias `processes`) split evenly | ignored | | `false` | one pool for all queues | scales between `minProcesses` and `maxProcesses` with the backlog | strict: each worker drains the first listed queue first | With `auto`, `minProcesses` is **per queue** and `maxProcesses` is the **total** across the supervisor's queues. With `false`, both are totals. ## Why auto starves billing Every `balanceCooldown` seconds (3 by default) the supervisor asks Horizon's auto-scaler for a target per queue: 1. It reads each queue's ready backlog from Redis. 2. It weighs the queues with the `autoScalingStrategy`: - `time` (default) - backlog multiplied by the queue's average job runtime, i.e. estimated time to clear; - `size` - the raw number of jobs waiting; - `log` - the logarithm of the job count (`log1p`), so a huge queue gets a less disproportionate share. 3. It multiplies each queue's share by `maxProcesses`. 4. It moves each pool toward its target by at most `balanceMaxShift` processes (1 by default) per cycle, never below `minProcesses` and never above the supervisor's total. A worked example with `maxProcesses => 10` and `minProcesses => 1`: billing has 20 jobs averaging 2 seconds (40 seconds of work), notifications has 40,000 jobs averaging 50 ms (2,000 seconds). Under `time`, billing's share is about 2%, so its target rounds up to one process, and notifications climbs to nine, the most it may take while billing keeps its floor. Under `size` the split is the same. Under `log` the weights are `log1p(20)` (about 3) and `log1p(40000)` (about 10.6), so billing's share rises to about a fifth and it holds two or three processes instead of one: better, but still load-driven, not guaranteed. Listing `billing` first changes nothing, because under `auto` the list order is not a priority. ## Fixes, in order of preference 1. **Split supervisors.** Give `billing` its own supervisor with, say, `minProcesses => 2` and `maxProcesses => 4`, and put `notifications` in a second supervisor with `maxProcesses => 6`. Each supervisor has its own budget, so a flood in one cannot take the other's processes. 2. **Strict priority with `balance => false`.** One pool works `['billing', 'notifications']` in that order: billing is always drained first and notifications wait. Use it only when starving the low-priority queue is acceptable. 3. **Fixed split with `simple`.** `processes => 10` over two queues gives five each: predictable, but idle billing workers cannot help notifications. 4. **Soften with `log`.** Keeps one supervisor but still lets the bigger backlog win; a mitigation, not a guarantee. Also raise `minProcesses` if a single billing worker cannot keep up with its steady rate on its own. ## The scale-down caveat When `auto` shrinks a pool, Horizon asks a worker to stop after its current job; if it is still running after the supervisor's `timeout` seconds, Horizon treats it as hanging and kills it. The docs therefore say to keep the supervisor `timeout` above any job-level timeout, so rebalancing never cuts a long billing job in half. ## Why this is a Horizon question Plain `queue:work` with `--queue=billing,notifications` gives strict order and a fixed process count set outside Laravel. Horizon's value is the dynamic middle ground, and the interview probe is whether you know that its default dynamic mode trades priority for throughput.

Under auto balancing, how quickly can Horizon react to a sudden backlog?
A queue's pool grows by at most `balanceMaxShift` processes per `balanceCooldown` seconds, 1 every 3 seconds with the published values, and never beyond the supervisor's remaining `maxProcesses`. Raising `balanceMaxShift` reacts faster but makes process churn more abrupt.
What does the time strategy need that the size strategy does not?
`time` multiplies each queue's backlog by the average runtime Horizon has recorded for that queue, so a few slow jobs can outweigh many fast ones. `size` counts jobs only. When no runtime has been recorded for any of the supervisor's queues, a queue with jobs is pushed toward `maxProcesses` and an empty one toward `minProcesses`.
Why can auto rebalancing kill a job halfway through?
A worker picked for scale-down is signalled to stop after its current job; if it is still running after the supervisor's `timeout` seconds Horizon treats it as hanging and stops it. Keep the supervisor `timeout` above your longest job-level timeout so rebalancing never interrupts real work.

saying these in an interview costs you the question

Listing billing first in the queue array gives it priority under auto
minProcesses under auto is a total for the whole supervisor
The simple strategy still scales workers with queue load
balance false stops Horizon from scaling the process count
The log strategy guarantees every queue an equal share of workers
Auto balancing jumps straight to its target process count

With Laravel Horizon, a burst of notification jobs leaves billing jobs waiting although one auto-balanced supervisor serves both queues; how do balance strategies explain this, and how would you fix it?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

The symptom

What each balance value does

Why auto starves billing

Fixes, in order of preference

The scale-down caveat

Why this is a Horizon question

▾common follow-ups3 questions

▾red flags6 flags

The symptom

What each balance value does

Why auto starves billing

Fixes, in order of preference

The scale-down caveat

Why this is a Horizon question

answer

go deeper