skip to content

How do Celery's worker_max_tasks_per_child and worker_max_memory_per_child stop report-rendering workers from growing, and what do they cost?

level: middleimportance: should knowfreq 35%

answer

  1. memory's high-water mark
  2. replace the process, not the task
  3. count versus resident size
  4. units are not megabytes

basics

~10 s

Both recycle Celery prefork child processes: worker_max_tasks_per_child replaces a child after N tasks, worker_max_memory_per_child (kilobytes) replaces it after the task that crossed the limit finishes. Each restart costs a fork and warm-up.

solid answer

~40 s

A Python process rarely returns freed memory to the operating system, so a child that once built a 3 GB DataFrame stays near that size. `worker_max_tasks_per_child = 50` (`--max-tasks-per-child`) retires each prefork child after 50 tasks and forks a fresh one. `worker_max_memory_per_child` (`--max-memory-per-child`) is in **kilobytes** of resident memory; it is checked after a task, so the task that crossed the limit completes and then the child is replaced — it never stops a running task. Both work only on `prefork`; other pools silently ignore them. The cost is a new process and its warm-up on every restart; set too low, children spend their time restarting. They cap slow growth and leaks, not one task that needs more RAM than the box has.

code

python · 9 lines
python
from celery import Celery

app = Celery('proj', broker='redis://localhost:6379/0')

# recycle a prefork child after 200 tasks...
app.conf.worker_max_tasks_per_child = 200

# ...or once it ends a task above ~2 GB resident (value is in KiB)
app.conf.worker_max_memory_per_child = 2_000_000

go deeper

for a junior

Recall that Celery can replace prefork child processes after a number of tasks or above a memory size, and that neither is on by default.

for a middle

Explain the high-water mark, the kilobyte unit, and why the memory check completes the current task before replacing the child.

for a senior

Pick limits from measured post-task memory, avoid restart churn, and recognise when a single oversized task needs a code fix rather than recycling.

for a principal

Treat recycling as a guard rail rather than a capacity plan, and weigh restart overhead against the cost of larger hosts or chunked task design.

## Why prefork children grow Under Celery's default **prefork** pool, each concurrency slot is a long-lived **child process** that runs task after task. Two things make those children grow: - **The high-water mark.** A Python process keeps most memory it has allocated until it exits. A report task that builds a large pandas DataFrame raises the child's resident size, and the child stays near that peak after the task ends. - **Leaks.** A library, often a C extension you cannot fix, keeps references between tasks, so memory climbs a little per task. Celery's optimizing guide makes the diagnosis split explicit: if the worker's **parent** process keeps growing, suspect a Celery bug; if only the **children** grow, the cause is in your tasks. ## The two recycling settings | Setting | CLI flag | Trigger | Unit | Default | |---|---|---|---|---| | `worker_max_tasks_per_child` | `--max-tasks-per-child` | child has run N tasks | task count | no limit | | `worker_max_memory_per_child` | `--max-memory-per-child` | child's resident memory exceeds the limit | kilobytes (1024 bytes) | no limit | When either triggers, the parent retires that child and forks a replacement; the other children keep working. ### worker_max_tasks_per_child A simple counter: after N tasks the child exits. It suits steady leaks where memory grows roughly per task. It does not look at memory at all, so a child that hits a huge report on task 3 of 50 stays large for the next 47. ### worker_max_memory_per_child Checks resident memory **after** each task. If a single task pushes the child past the limit, that task is **completed** and the child is replaced afterwards. Two traps follow: 1. **Units.** The value is kilobytes. `worker_max_memory_per_child = 2000` is about 2 MB — every child exceeds it after its first task and is replaced every time. For roughly 2 GB write `2000000`. 2. **Timing.** It cannot stop a task mid-flight. One report that needs 12 GB on a 16 GB host still exhausts memory; the kernel's OOM killer may kill the child, and the task fails as lost. ## Pool support Both settings are marked **prefork only** in the workers guide, and the concurrency guide notes that switching pools silently disables features such as `max_tasks_per_child`. On `gevent`, `eventlet`, `threads` or `solo` there is no child to replace, so the settings have no effect — another reason report rendering belongs on prefork. ## What recycling costs - **Fork and warm-up.** Each new child re-runs process initialisation, re-opens connections and repopulates caches. - **Throughput ceiling.** The optimizing guide's example: with `worker_max_tasks_per_child = 1` and a child that takes one second to start, that slot processes at most 60 tasks a minute. - **Churn from a low memory limit.** If normal tasks always exceed `worker_max_memory_per_child`, you get the same restart-per-task behaviour. ## Choosing between them - Memory climbs a little on **every** task (a steady leak): `worker_max_tasks_per_child` is predictable and cheap to reason about. - Memory jumps on **some** tasks (a big report now and then): `worker_max_memory_per_child` replaces only the children that actually grew. - Both patterns at once: set both; whichever triggers first retires the child. Neither setting changes how many tasks run at once or how many messages are prefetched; they only decide when a slot's process is swapped for a fresh one. ## A reasonable setup for report workers 1. Measure a child's resident memory after typical and worst-case reports. 2. Set `worker_max_memory_per_child` in kilobytes: above the typical post-task size when tasks are short, so children do not churn, or below the worst peak when tasks run for minutes and a restart is cheap by comparison. 3. Add a generous `worker_max_tasks_per_child` (for example a few hundred) if a library leaks steadily. 4. For single tasks that are simply too big, fix the task — chunk the spreadsheet, stream rows — or lower `-c` so peak × concurrency fits in RAM.

  • A team sets Celery's worker_max_memory_per_child = 500 meaning 500 MB; what happens?
    The unit is kilobytes, so the limit is about 500 KB. Every child exceeds it after its first task and is replaced after every task, which behaves like `worker_max_tasks_per_child = 1` with a fork each time. The intended value is about `500000`.
  • Would worker_max_memory_per_child save a Celery worker from one report that needs more RAM than the host has?
    No. The check runs after a task completes, so it never interrupts the task. The oversized report can still exhaust memory, and the OS may kill the child mid-task. Fix the task's peak memory or lower concurrency instead.

Like replacing a whiteboard marker after a set number of pages, or when it starts to smear: you swap it between pages, never mid-sentence, and swapping too often wastes more time than the smudges did.

saying these in an interview costs you the question

  • worker_max_memory_per_child is measured in megabytes.
  • worker_max_memory_per_child kills a task the moment memory crosses the limit.
  • Setting max_tasks_per_child on a gevent worker recycles its greenlets.
  • worker_max_tasks_per_child = 1 has no throughput cost.
  • Recycling children fixes memory growth in the worker's parent process.