A Prefect task calling a flaky API fails intermittently. How do you configure retries and timeouts?

## Where retries are configured The task run is the retry boundary, so the knobs live on the decorator: ```python from prefect import task from prefect.tasks import exponential_backoff @task( retries=3, retry_delay_seconds=exponential_backoff(backoff_factor=10), retry_jitter_factor=1, timeout_seconds=60, ) def fetch_page(page: int) -> dict: ... ``` `retries=3` means up to three additional attempts after the first, four executions worst case. Between attempts the run sits in a retrying state and the same task run is reused, so its identity and logs are continuous rather than a fresh run per attempt. ## Shaping the delay `retry_delay_seconds` accepts three shapes and choosing between them is a real answer: - **A single number** — `retry_delay_seconds=5`. Fine for a service that recovers in seconds. - **A list** — `retry_delay_seconds=[1, 10, 60]`. One entry per attempt, which is the most explicit form: fast first retry for a blip, long final retry for a deploy-shaped outage. The list length should match `retries`. - **`exponential_backoff(backoff_factor=n)`** — generates a growing sequence, the standard choice against a rate-limited or overloaded dependency. Add `retry_jitter_factor` on top. It randomises each delay, and it matters precisely in the fan-out case this tree cares about: 200 mapped task runs all hit the same API, all get throttled at the same instant, and all retry at the same instant unless the delays are spread. Lockstep retries turn a small overload into a self-sustaining one. ## Not every failure deserves a retry Retrying a deterministic failure is pure waste: a 400 from a malformed payload, a schema mismatch, a missing permission or an assertion error will fail identically four times and simply delay the alert by the sum of the backoff. Distinguishing transient from deterministic is the senior half of this question. `retry_condition_fn` lets you encode it — it receives the task, the task run and the state, and returns whether to retry — so you can retry a 429 or 503 and fail fast on a 4xx that will never change. If you cannot use it, raise a distinct exception type for permanent failures and keep the retriable path narrow. ## Timeouts `timeout_seconds` is available on tasks and on flows, and it is the knob most often missing in a real incident. The failure mode is specific: a request with no client-side timeout hangs on a half-open socket. The task run never fails, so `retries` never fires — retries are triggered by failure, and a hang is not a failure. The run occupies a worker slot forever, the flow run never completes, and if the flow is scheduled you accumulate overlapping runs. A timeout converts a hang into a failure, which then interacts correctly with everything else: it becomes visible, it can be retried, and the slot is released. Set it deliberately — comfortably above the observed p99 of the operation, not at the mean, or you will manufacture failures on ordinary slow days. Belt and braces: also set the client library's own timeout, because that fails faster and closer to the cause. Flow-level `timeout_seconds` is a different guarantee: it bounds the whole flow run, catching the case where no single task hangs but the run as a whole drags past its useful window. ## Task retries versus flow retries Both exist, and confusing them is a red flag. `@task(retries=...)` re-runs one unit of work. `@flow(retries=...)` re-runs the flow function — in general re-executing its steps, which is far heavier and only appropriate when the whole stage is safely repeatable. The default instinct should be task-level retries for transient dependency faults, and flow-level retries reserved for a stage that is genuinely idempotent end to end. ## The idempotency condition A retry is a second execution of side effects. If `fetch_page` only reads, retrying is free. If the task charges a card, sends an email, or appends rows to a table, the retried attempt repeats it, and a flaky network on the *response* path — where the write succeeded but the acknowledgement was lost — makes duplicates especially likely. Make the write safe to repeat: an idempotency key on the API call, a merge or upsert keyed on a natural identifier, writing to a deterministic path that overwrites, or a check-then-act guarded by a unique constraint. "Add retries" without asking this question is how a retry policy quietly becomes a data-quality incident. ## Observability Retries hide instability. A pipeline whose tasks quietly retry twice every night is failing constantly and nobody notices until the third attempt starts failing too. Watch attempt counts, not only final states, and treat a rising retry rate as a signal about the dependency. ## How to answer Name the knobs precisely, explain backoff plus jitter and why jitter matters for a fan-out, insist on `timeout_seconds` because retries cannot rescue a hang, separate transient from deterministic failures, and finish on idempotency. That ordering is the difference between reciting parameters and demonstrating you have run this in production.