Why does calling AsyncResult.get() inside a Celery task raise RuntimeError, and how should a thumbnail workflow wait for another task instead?
answer
- a worker slot that only waits
- pool exhaustion
- disable_sync_subtasks
- make the next step a callback
basics
~20 sCelery's AsyncResult.get() defaults to disable_sync_subtasks=True, so inside a worker task it raises RuntimeError: a task blocking on another can exhaust the pool and deadlock. Express the dependency with a chain, chord or link callback instead.
solid answer
~40 sA task that calls `.get()` on another task's result holds its worker slot while doing nothing. If every slot fills with parents waiting on children that are still queued, no slot is left to run the children and the workers deadlock. So Celery forbids it by default: `AsyncResult.get()` takes `disable_sync_subtasks=True`, and when the worker's pool is a blocking one (prefork, threads, solo) the call raises `RuntimeError: Never call result.get() within a task!` before waiting at all; a `timeout` does not change that. The fix is to make the dependency part of the workflow: `make_thumbnail.s(photo_id, 640) | publish_photo.s(photo_id)` as a chain, a chord for several thumbnails, or a `link` callback, so the next task is sent only when the result exists. `get(disable_sync_subtasks=False)` exists as an escape hatch, but Celery's docs warn against it.
code
python · 23 linesfrom celery import Celery
app = Celery('photos', broker='redis://localhost:6379/0',
backend='redis://localhost:6379/1')
@app.task
def make_thumbnail(photo_id, size):
return f'thumbs/{photo_id}_{size}.jpg'
@app.task
def publish_photo(thumb_path, photo_id):
return photo_id
@app.task
def handle_upload_bad(photo_id):
# prefork worker: RuntimeError: Never call result.get() within a task!
path = make_thumbnail.delay(photo_id, 640).get(timeout=30)
publish_photo.delay(path, photo_id)
@app.task
def handle_upload(photo_id):
# sends the workflow and returns; nothing waits
(make_thumbnail.s(photo_id, 640) | publish_photo.s(photo_id)).delay()go deeper
Recall that a Celery task must not wait on another task's result, and that chains or callbacks replace the wait.
Explain disable_sync_subtasks, the RuntimeError, the pool-exhaustion deadlock step by step, and how a chain or chord expresses the same dependency.
Show you can spot hidden waits, such as ready() polling or a helper calling get(), and justify when an isolated queue plus disable_sync_subtasks=False is tolerable.
Discuss the design stance: workflows described as data that the queue advances, versus tasks that orchestrate by blocking, and what that means for capacity planning.
## The rule and the error Celery's `AsyncResult.get()` waits for a task's result and returns it. Called from ordinary code, such as a script or a web request, that is fine. Called from **inside a running task**, it raises immediately: ``` RuntimeError: Never call result.get() within a task! ``` The guard comes from the `disable_sync_subtasks` parameter of `get()`, which defaults to `True`. When it is true, `get()` first calls an internal check that raises if the current process is a worker whose pool would block on the wait. `GroupResult.get()` and `join()` carry the same parameter and the same check. The check runs **before** any waiting, so passing `timeout=30` does not help: the call fails at once. ## Why waiting inside a task is dangerous A Celery worker runs tasks in a fixed number of **pool slots**, set by `--concurrency`. A task that calls `.get()` keeps its slot while it waits. Consider an upload task that sends `make_thumbnail` and then waits for it: 1. Four uploads arrive on a worker with four slots. 2. Each upload task sends a `make_thumbnail` task and blocks in `.get()`. 3. The four thumbnail tasks sit in the queue, but every slot is held by a waiting parent. 4. No slot frees up, so no thumbnail runs, so no parent finishes: a **deadlock**. More concurrency only moves the threshold: a burst of uploads larger than the slot count reproduces it. Even without a deadlock, a waiting slot is a wasted slot, and the result backend is polled or subscribed to for nothing. ## Which pools raise The guard depends on the pool, because each pool declares whether a join would block: | Pool | `get()` inside a task | |---|---| | `prefork` (default) | Raises `RuntimeError` | | `threads`, `solo` | Raises `RuntimeError` | | `gevent`, `eventlet` | Does not raise; the green pools set the flag off | On the green pools the wait yields to other greenlets instead of freezing a process, so Celery does not stop it. The design problem is unchanged, though: the waiting task still occupies a slot and still ties its completion to a queue it does not control. ## The fix: let the workflow wait The asynchronous alternative is to describe the dependency so that Celery sends the next task **when the result exists**, and no task ever waits: - **chain**: `make_thumbnail.s(photo_id, 640) | publish_photo.s(photo_id)` sends `publish_photo` only after the thumbnail succeeds, with its result prepended. - **chord**: for several sizes, a chord runs the thumbnails in parallel and then `publish_photo` once with the list of paths. - **link**: `make_thumbnail.apply_async((photo_id, 640), link=publish_photo.s(photo_id))` attaches the same callback to a single call. The same applies to groups: a task that sends a group and then calls `.get()` on its `GroupResult` hits the same guard. Make the follow-up the **body of a chord** instead, and let the join happen in the result backend rather than in a waiting task. The upload task's job becomes *building and sending* the workflow, then returning. Each step runs when its input is ready and gives its slot back immediately. ## The escape hatch and why it stays shut Celery does let you override the guard: - `result.get(disable_sync_subtasks=False)` for one call; - the `allow_join_result()` context manager from `celery.result`, which lifts the guard for a block of code. The docs mark this as not recommended. If you must use it, the child tasks need their **own queue and workers**, so parents can never occupy the slots the children need, and every wait needs a `timeout`. Celery itself uses `allow_join_result()` internally when its built-in chord machinery joins a header whose tasks have already finished; that is a join over results that already exist, not a wait for queued work. ## What interviewers are checking - That you know the call **raises by default** and why, not just that it is "slow". - That you can describe the **pool-exhaustion deadlock** step by step. - That your fix is a **callback-shaped workflow** (chain, chord or link), not polling `result.ready()` in a loop, which blocks the slot just the same while dodging the guard. - That you treat `disable_sync_subtasks=False` as a last resort with isolation, not a fix.
- Why doesn't the same .get() call raise when the Celery worker runs the gevent pool?Each pool declares whether a join would block. The `gevent` and `eventlet` pools set that flag off, because a waiting greenlet yields instead of freezing a process, so `assert_will_not_block()` never fires. The call then works, but the waiting task still holds one of the pool's slots, and a burst of waiting parents can still starve the children they wait for.
- When, if ever, is get(disable_sync_subtasks=False) acceptable in a Celery task?Only as a deliberate exception: the child runs on a separate queue served by separate workers, so a waiting parent cannot occupy the slot its child needs, and the wait carries a `timeout`. Even then a chain, chord or link is almost always simpler. Celery's docs label enabling synchronous subtasks as not recommended.
saying these in an interview costs you the question
- Passing a timeout to .get() makes it safe to call inside a task.
- The RuntimeError means the result backend is misconfigured.
- Polling result.ready() in a loop is a safe replacement for .get().
- Raising worker concurrency removes the deadlock risk for good.
- Celery raises because the child task has not been sent yet.