skip to content

In Celery, how do link and link_error callbacks behave, and what does a link_error errback receive when a thumbnail task fails?

level: middleimportance: should knowfreq 22%

answer

  1. success path versus failure path
  2. return value goes in front
  3. request, exc, traceback
  4. groups hand links to members

basics

~20 s

In Celery, a link callback is sent only after the task succeeds, with its return value prepended to the callback's arguments. A link_error errback runs when the task fails; a new-style errback receives the request, the exception and the traceback.

solid answer

~50 s

`link` and `link_error` attach signatures to a task call, either as `apply_async(link=..., link_error=...)` or through the signature methods `.link()`, `.link_error()` and the chainable `.on_error()`. A `link` callback is applied only when the task **succeeds**, as a new task with the return value prepended to its arguments; several callbacks are sent together, each receiving that value. A `link_error` errback fires when the task **fails**. If the errback task takes more than one parameter, the worker calls it directly with the failed task's `request`, the exception and the traceback; an old-style errback taking a single argument is sent as a task with just the failed task id. On a `group`, links behave unexpectedly: `group.link()` attaches the callback to the first member only, while `group.link_error()` copies the errback to every member, so it can run once per failing thumbnail.

code

python · 22 lines
python
from celery import Celery

app = Celery('photos', broker='redis://localhost:6379/0')

@app.task
def make_thumbnail(photo_id, size):
    return f'thumbs/{photo_id}_{size}.jpg'

@app.task
def publish_photo(thumb_path, photo_id):
    return photo_id

@app.task
def log_thumb_failure(request, exc, traceback):
    # called directly by the worker, not queued
    print(f'thumbnail {request.id} for {request.args} failed: {exc!r}')

make_thumbnail.apply_async(
    (42, 640),
    link=publish_photo.s(42),          # publish_photo(<path>, 42) on success
    link_error=log_thumb_failure.s(),  # on failure only
)

go deeper

for a junior

Recall that link runs a follow-up task after success and link_error runs an error handler after failure.

for a middle

Explain the prepended return value, the (request, exc, traceback) errback signature, and the difference between link_error() and on_error().

for a senior

Show you know where callbacks surprise: group.link hits one member, group and chord-header errbacks repeat, and in-worker errbacks must stay fast and idempotent.

for a principal

Discuss where failure handling belongs in a workflow: in-worker errbacks, task-level retries, or an external state machine, and how each is observed.

## Two callback slots on every task call Every Celery task call can carry two lists of **signatures** that run after it: - **`link`**: callbacks for the **success** path. - **`link_error`**: errbacks for the **failure** path. You set them when sending, `make_thumbnail.apply_async((42, 640), link=publish_photo.s(42), link_error=log_thumb_failure.s())`, or on a signature before it is sent: | Method | Effect | Returns | |---|---|---| | `sig.link(cb)` | Adds a success callback | The callback `cb` | | `sig.link_error(eb)` | Adds an errback | The errback `eb` | | `sig.on_error(eb)` | Adds an errback | The original `sig`, so you can keep chaining | The return values matter when you write `make_thumbnail.s(42, 640).on_error(eb).delay()`: that sends the thumbnail task. Writing the same with `link_error(eb)` would try to send the errback. ## The success path: link A **link callback** is applied only after the parent task **succeeds**: 1. The worker finishes the task and stores its return value. 2. It sends the linked signature with the return value as a **partial argument**; with several callbacks, it sends them together as one group, each receiving the same value. 3. Partial arguments are **prepended**, so `publish_photo.s(42)` runs as `publish_photo('thumbs/42_640.jpg', 42)`. If the callback should not take the result, link an immutable signature, `notify_owner.si(1001)`. A failed or retried-until-failure task never triggers its `link` callbacks. A chain follows the same rule through its own bookkeeping: the next step is sent only after success, with the result prepended. ## The failure path: link_error An **errback** runs when the task **fails**. Celery supports two errback styles, and it picks one by looking at the errback task's parameters: - **New style** (more than one parameter), for example `def log_thumb_failure(request, exc, traceback)`. The worker does **not** send it as a task; it calls the function **directly**, in the worker process that handled the failure, passing the failed task's `request` context, the exception instance and the traceback. Any arguments bound on the errback signature come after those three. - **Old style** (a single parameter). It is sent as a normal task message with the failed task's **id** as its argument. Because a new-style errback runs inside the worker that just failed, it should be quick: a slow errback delays that worker. The `request` gives it `request.id`, `request.args` and `request.kwargs`, which is usually enough to log the failure or mark the upload as broken. ## Groups and chords change the picture A group is not a real task, so it passes links down to its members, with surprising results: - **`group.link(cb)`** links the callback to the **first** task in the group only, cloned as immutable. It can run before the other members finish and never sees their results. - **`group.apply_async(link=...)`** raises `TypeError` and tells you to use a chord. - **`group.link_error(eb)`** clones the errback onto **every member**, so if two of five thumbnails fail, it runs twice. For "run once after all of them", use a **chord**. A chord's `link()` goes to its body; its `link_error()` goes to the body only, unless `task_allow_error_cb_on_chord_header` is enabled. On a **chain**, an errback linked to the chain is attached to every step, so whichever step fails triggers it. ## link or chain? Both send a follow-up task on success with the result prepended, so the choice is about shape. `link` attaches a callback to **one call you are already sending**; it suits a single follow-up or several independent ones. A `chain` describes the **whole sequence up front**, returns the last step's `AsyncResult` with `.parent` links back through the steps, and reads better once there are three or more steps. ## Applying it to the upload pipeline - Use `link` for a single follow-up on success, such as publishing after one thumbnail. - Use `link_error` or `on_error()` to mark the upload failed and alert, with a new-style errback that reads `request.args` to find the photo. - Make errbacks **idempotent**, since group and chord-header errbacks may fire more than once per workflow. - Keep errbacks to logging and state changes; retrying the failed step belongs to the task's own retry options. ## What interviewers probe - That `link` fires **only on success** and passes the return value **first**. - That new-style errbacks receive `(request, exc, traceback)` and run **in the worker**, not as a queued task. - That `group.link()` is not a join, and that group errbacks can repeat.

  • What is the practical difference between sig.link_error(eb) and sig.on_error(eb) in Celery?
    Both add `eb` to the signature's `link_error` list. `link_error()` returns the errback, while `on_error()` returns the original signature. So `make_thumbnail.s(42, 640).on_error(eb).delay()` sends the thumbnail task, but the same expression with `link_error(eb)` would call `delay()` on the errback.
  • Why is group.link(callback) not a way to run something after every Celery group member finishes?
    A group is not a task, so it links the callback to its first member only, cloned as immutable. The callback can fire before the other members finish, even if they later fail, and never receives their results. Passing `link=` to `group.apply_async()` raises `TypeError` pointing you to a chord, which is the primitive that joins.

saying these in an interview costs you the question

  • A link callback runs whether the task succeeds or fails.
  • A new-style errback is queued as a normal task on any worker.
  • An errback linked to a group runs once, however many members fail.
  • group.link(callback) runs the callback after every group member finishes.
  • link_error() and on_error() return the same object, so they chain identically.