A Celery worker busy with long report tasks holds short webhook tasks while another worker sits idle; how do worker_prefetch_multiplier and -O fair explain it?

## What prefetch means in Celery **Prefetch** is the number of messages a worker may reserve from the broker before it has a free slot to run them. The broker delivers messages up to that limit; they stay **unacknowledged** and live in the worker's memory as **reserved** tasks. Until the worker acknowledges them or its connection drops (or, on Redis and SQS, their visibility timeout expires), no other worker can receive them. Celery computes the limit as: - `worker_concurrency × worker_prefetch_multiplier`; - `worker_prefetch_multiplier` defaults to **4**; - a multiplier of **0** removes the limit entirely — the worker keeps consuming as many messages as it wants. Prefetching exists for throughput: a worker running thousands of 20 ms tasks should not pay a broker round-trip before each one. ## How it starves short tasks Take two workers, A and B, each with `-c 4` on the `prefork` pool, both consuming one queue that mixes 10-minute report renders with 200 ms webhook calls. | Moment | Worker A | Worker B | |---|---|---| | A starts first | reserves 16 messages: 4 reports, 12 webhooks | not yet connected | | A's 4 children run reports | 12 webhooks wait in A's buffer | idle, queue is empty | | 10 minutes later | webhooks finally run | still idle | The configuration docs warn about exactly this: the first worker to start receives four times its concurrency in messages, so long tasks are not fairly distributed. ## What -O fair actually does `-O` / `--optimization` accepts `default` or `fair`. In the prefork pool the `default` profile **maps to the fair scheduling strategy**, and has done since Celery 4.0. The fair strategy is about dispatch **inside one worker**: the parent process will not write a task into the pipe of a child that is already executing one, so a short task is not stuck behind a long one when another child of the same worker is free. It does nothing about messages **reserved across workers**. Advice to "add `-Ofair`" predates 4.0; on 5.6 it changes nothing. ## The fixes and what each costs | Setting | Effect | Cost | |---|---|---| | `worker_prefetch_multiplier = 1` | at most one extra reserved message per slot (early acks) | more broker round-trips for short tasks | | multiplier 1 + `task_acks_late = True` | reserves only as many as there are slots | tasks must be idempotent; a killed worker's task reruns | | `worker_disable_prefetch = True` (5.6) | fetches only when a slot is free, keeps early acks | Redis broker only; other transports ignore it with a warning | | separate queues and workers | long and short tasks never share a buffer | a second deployment to run | The optimizing guide's own recommendation for a mix of long and short tasks is the last row: two worker nodes configured separately, with tasks routed by run time. The report workers then run multiplier 1; the webhook workers can keep a larger multiplier because their tasks are short. ## ETA tasks slip past the window Tasks sent with `countdown` or `eta` are received immediately and held on the worker's internal timer until due. The worker raises its prefetch allowance for each one so they do not block normal work, which is why `--prefetch-multiplier=1` can appear to have no effect when many retried webhooks carry countdowns. Celery **5.6** adds `worker_eta_task_limit` (default `None`, no limit) to cap how many ETA tasks a worker holds; once the cap is reached it stops consuming until some run. ## Confirming it in production 1. Run `celery -A proj inspect reserved` and look for many webhook tasks reserved on a worker whose slots are all busy. 2. Compare with `inspect active` on the idle workers. 3. Check the startup banner or `app.conf` for the effective multiplier and concurrency. ## Why raising concurrency is not the fix A tempting response is to add slots to the busy worker. On `prefork`, that also multiplies the reservation: going from `-c 4` to `-c 8` with the default multiplier raises the prefetch window from 16 to 32, so the worker hoards more webhooks behind more reports, and every extra child costs a full process's memory. The starvation comes from **mixing run times in one buffer**, not from too few slots, so the lasting fixes are the ones above: - shrink the buffer for long tasks (multiplier 1, or no prefetch); - or stop long and short tasks from sharing a buffer at all (separate queues and workers).