skip to content

In Celery, what are task events and the worker's -E flag, and what does Flower need from the workers to show a task's history?

level: middleimportance: should knowfreq 30%

answer

  1. a live feed, not the result backend
  2. worker events on, task events off
  3. Flower rebroadcasts enable_events
  4. in memory unless persistent
  5. no events on SQS

basics

~20 s

Task events are messages a Celery worker publishes as each task is received, started, succeeds or fails. They are off by default; celery worker -E turns them on. Flower builds its task list from that stream, so without events it shows no task history.

solid answer

~40 s

Workers can publish **events** such as `task-received`, `task-started`, `task-succeeded`, `task-failed` and `task-revoked`, plus `worker-heartbeat` messages. Heartbeats flow by default, but task events are off (`worker_send_task_events=False`) unless the worker runs with `-E` or receives the `enable_events` remote-control command; `task-sent` also needs `task_send_sent_event=True` on the producer. Flower (`celery -A proj flower`) is an event consumer: its task list holds only what it has seen since it started, in memory, unless it runs with `--persistent=True`. Flower broadcasts `enable_events` to workers every 5 seconds by default, which is why it often works without `-E`, but that depends on remote control, and a freshly started worker is silent until the next broadcast. On SQS neither events nor remote control exist. Start workers with `-E` so monitoring does not hinge on Flower.

code

bash · 8 lines
bash
# workers publish task events from the moment they start
celery -A proj worker -E -l INFO

# Flower keeps its task history across its own restarts
celery -A proj flower --port=5555 --persistent=True --db=/var/lib/flower/flower.db

# the raw event stream in a terminal
celery -A proj events --dump

go deeper

for a junior

Recall that Flower is a web monitor fed by Celery events, and that workers need task events switched on, with -E, for tasks to appear.

for a middle

Explain the difference between events and the result backend, which events are on by default, and how Flower's periodic enable_events broadcast hides a missing -E.

for a senior

Know where monitoring goes blind: worker restarts before the next broadcast, remote control disabled, SQS, Flower restarts without persistence. Set -E explicitly and lock Flower down.

for a principal

Decide whether Flower is enough or task events should feed a durable store, weighing the extra broker traffic of events against the audit and incident needs of the team.

## Two different streams Celery has two ways to learn what happened to a task, and they are easy to confuse: - The **result backend** stores one record per task id, its state and return value, for callers that hold an `AsyncResult`. - **Events** are small messages that workers, and optionally clients, publish to the broker as things happen: a task was received, started, succeeded, failed, retried or revoked; a worker came online, sent a heartbeat or went offline. They are a live feed for monitors, not stored history. Flower, `celery -A proj events` and any custom monitor built on `app.events.Receiver` consume the second stream. Events do not depend on the result backend, and `task_track_started`, which makes the backend record `STARTED`, has no effect on them: the worker sends `task-started` whenever task events are on. ## What is on by default | Events | Default | How to turn on | |---|---|---| | Worker events: `worker-online`, `worker-heartbeat`, `worker-offline` | Sent | On unless switched off with `--without-heartbeat` and `--without-gossip` | | Task events from workers: `task-received`, `task-started`, `task-succeeded`, `task-failed` and others | Off, `worker_send_task_events=False` | `celery -A proj worker -E`, the setting, or the `enable_events` control command | | `task-sent` from the producer | Off, `task_send_sent_event=False` | `task_send_sent_event=True` in the client's configuration | So a worker started plainly announces itself but says nothing about the tasks it runs. That is the classic picture of a monitor that lists workers but no tasks. ## How Flower uses them `celery -A proj flower` (port 5555 by default) connects to the broker, subscribes to events, and builds its dashboard from them: 1. Each event updates an in-memory state: task name and arguments from `task-received`, runtime from `task-succeeded`, the exception from `task-failed`. 2. The task list keeps at most `max_tasks` entries, 100,000 by default, and covers only what Flower has seen **since it started**. 3. With `--persistent=True` Flower saves that state to its `--db` file and reloads it on restart; otherwise a Flower restart begins with an empty list. 4. Its worker pages and action buttons (revoke, pool resize, time limits) use remote control, not events. Flower also has an `enable_events` option, on by default: every 5 seconds it broadcasts the `enable_events` control command to all workers. That is why many teams never pass `-E` and still see tasks: Flower is switching task events on for them. For the hung PDF export, Flower's task page then shows a task that was received and started with no finishing event, the worker it is on, and its arguments, which is usually enough to find the export and the font request behind it. ## Where the picture breaks - **Worker restarts.** A worker replaced during a rolling deploy starts with task events off if it has no `-E`; until Flower's next broadcast its tasks are invisible. The gap is short, but a quick task can come and go unseen. - **Remote control off.** With `worker_enable_remote_control=False`, Flower's broadcast never lands, so only workers that enabled task events at start, with `-E` or `worker_send_task_events`, report tasks. - **Amazon SQS.** Celery's SQS documentation states that SQS supports neither events nor remote control. `-E` does not help there, and Flower cannot show those workers' tasks. - **Flower down.** Events are a live feed: Flower does not replay what was published while it was not listening. ## A setup that holds up 1. Start workers with `-E`, or set `worker_send_task_events=True`, so task events never depend on Flower being up. 2. Set `task_send_sent_event=True` on producers if you want to see tasks from the moment they are sent, not only once a worker receives them. 3. Run Flower with `--persistent=True` if its history must survive its own restarts, and still treat it as a monitoring view rather than an audit log. 4. Protect it. Flower can revoke tasks and shut down workers, so put it behind `--basic_auth` or its OAuth support, or run it in `read_only` mode where viewing is all anyone needs. ## The cost of events Every task produces several extra broker messages, at least received, started and a final one. For high-volume short tasks that is measurable broker traffic, which is why task events are opt-in. A PDF-export workload, minutes per task at modest volume, is the easy case: leave them on, and the hung export shows up in Flower as a task that was received and started but never finished.

  • Does task_track_started have to be on for Flower to show when a task started?
    No. The worker sends a `task-started` event whenever task events are enabled, and Flower reads that. `task_track_started` only decides whether the result backend records a `STARTED` state for callers polling an `AsyncResult`; it is a separate channel with a separate default of False.
  • Why do Flower's counts drop to zero after Flower itself restarts?
    Flower keeps its task state in memory, built from events it received since it started. Events are a live feed, not a log, so nothing replays the past. Running Flower with `--persistent=True` and a `--db` file saves and reloads its state, which covers Flower's own restarts but not the time it was down.

saying these in an interview costs you the question

  • Flower reads task history from the result backend, so events are optional
  • task_track_started must be on for Flower to see tasks start
  • The broker stores events, so Flower can replay history after a restart
  • Starting workers with -E makes Flower work on the SQS transport
  • celery worker -E also makes producers emit task-sent events