skip to content

Event Delivery & Retries

Between an event source and a function sit retry counts, batch windows, timeouts and failure destinations. Interviewers ask how you would tune them and replay what failed.

on this pageshow

questions

6

Why does an asynchronously invoked function need a failure destination when the function platform already retries failed events?

level: juniorimportance: must knowfreq 65%

answer

  1. the caller stopped waiting long ago
  2. retries are bounded, not endless
  3. out of attempts or too old
  4. keep the event and its error
  5. alarm on anything arriving there

basics

~20 s

Asynchronous retries are bounded: when they run out or the event ages past its limit, the platform discards it, and the caller left long ago. A failure destination keeps the event and its error for alarming, inspection and replay.

solid answer

~50 s

With asynchronous invocation the platform accepts the event, tells the caller "accepted" straight away and queues the event internally. If the handler fails, the platform retries a small, configurable number of times with a delay between attempts, and it also gives up on any event older than a configured **maximum event age**. When either limit is reached the event is dropped, and because the caller only ever saw "accepted", nothing upstream knows. A **failure destination** (a queue, a topic, an event bus or another function) receives a record of every event that ran out of attempts, usually with the error, the attempt count and timestamps. That turns silent loss into something you can alarm on, debug and replay once the cause is fixed. For messages a queue redelivers on its own, the catch-net is the queue's dead-letter queue, not the function's setting.

go deeper

for a junior

Remember the chain: accepted, retried a few times, then dropped unless a failure destination is configured. The caller never hears about the failure.

for a middle

Explain the two limits that end an event's retries, retry count and maximum event age, and what the destination record carries so the event can be replayed.

for a senior

Show you place the catch-net where the retries happen, alarm on arrivals with an owner, and size retention to outlast the longest stretch nobody is watching.

for a principal

Frame a failure destination as a fleet default rather than a per-team choice, and weigh its cost, alarm noise and storage, against silent loss of business events.

## What asynchronous invocation does with an event On a function platform an event source can hand an event over in two broad ways. In the **synchronous** path the caller waits for the function's result and sees any error itself. In the **asynchronous** path the platform accepts the event, answers the caller with a plain acknowledgement ("accepted") and puts the event on an internal queue. A worker inside the platform then invokes the function for that event, and the caller has already moved on. That handoff is what makes asynchronous invocation useful: an upload notification, an order-placed event or a scheduled job does not need to wait for the work. It is also what makes failure dangerous. The only party that knows the event failed is the platform. ## Where an event goes when the retries run out When the handler throws or times out on an asynchronous event, the platform does not give up at once. It retries, with a growing delay between attempts. Two settings bound that loop: - **Retry count** - how many extra attempts the platform makes after the first failure. It is configurable, usually from zero up to a small maximum. - **Maximum event age** - how long the platform keeps trying from the moment it accepted the event. Time spent waiting in the internal queue (for example while the function is throttled) counts toward it. Whichever limit is reached first ends the event's life. What happens next depends on one setting: 1. With **no failure destination**, the event is discarded. A function error metric may show that attempts failed, but the event itself and its payload are gone. 2. With a **failure destination**, the platform writes a record of the event there and then stops. So the platform's retries cover **transient** trouble: a brief network blip, a downstream service that restarts within the retry window. They do not cover an outage longer than the retry window, a bug that fails every attempt or a malformed payload. Those are exactly the cases where you most need the event back. ## What a failure destination receives The destination is a durable target the operator chooses: a queue, a pub/sub topic, an event bus or another function. The record it receives typically contains: - the **original event payload**, so it can be replayed unchanged; - **error details** from the last attempt (error type, message, sometimes the function's response); - the **attempt count** and timestamps, so you can tell a slow failure from an instant one; - the **function identity** (name and version or alias), so a shared destination can be sorted by owner. Exact fields differ between platforms, so check what yours includes before you build replay tooling on it. Some platforms also offer a **success destination** on the same path, used to chain work rather than to catch failures. ## Function-side destination vs source-side dead-letter queue The catch-net belongs where the retries happen. A function's failure destination covers events the platform invoked asynchronously. A queue that the platform polls on the function's behalf redelivers messages itself, counts how often each one was received, and moves a message to **its own dead-letter queue** after a maximum receive count. | Event reaches the function through | Who retries | Where failed events land | |---|---|---| | The platform's asynchronous invocation path | The function platform | The function's failure destination | | A queue the platform polls | The queue, by redelivery | The queue's dead-letter queue | | A pub/sub subscription that pushes to the function | The subscription's delivery policy | The subscription's dead-letter target | Configuring the wrong side leaves a gap: a destination on the function does nothing for a message that a queue keeps redelivering. ## Making the destination useful A destination nobody looks at is only a slower way to lose events. The operating habits that make it work: - **Alarm on arrivals.** One message in a failure destination means at least one event ran out of attempts. Alarm on any arrival or a non-zero depth, not on a dashboard someone might open. - **Name an owner.** The alarm goes to the team that owns the function, not to a shared inbox. - **Keep retention longer than your blind spot.** If the team can be unavailable over a weekend, the destination must keep records well past that, or they expire unseen. - **Plan the replay.** Know how a record is turned back into an event and sent through the normal path after the fix. A function error rate is not a substitute: it counts failed attempts, including ones a later retry fixed, and it never tells you which payloads were finally lost.

  • What should alert on a failure destination, and why is a dashboard not enough?
    Alarm on any arrival or a non-zero depth, and on the age of the oldest record, routed to the function's owning team. A dashboard needs someone to look at it, and a destination's retention is finite: records that arrive on a Friday night and are noticed after they expire are lost exactly as if no destination existed.
  • If a function reads from a queue the platform polls for it, does the function's failure destination catch a message that keeps failing?
    Usually not. The queue redelivers the message itself after each failed attempt and counts receives; once the maximum receive count is reached the queue moves it to its own dead-letter queue. That queue-side setting is the catch-net there. Check, per source, which side does the retrying and put the catch-net on that side.

A courier who tries a delivery a few times and then returns the parcel to a depot with a note saying why. Without the depot, the parcel is simply thrown away after the last attempt, and the sender, who was told "accepted for delivery", never hears about it.

  • The platform keeps retrying a failed asynchronous event until it eventually succeeds.
  • The caller receives the error, so it can retry the asynchronous invocation itself.
  • The function's logs hold enough to rebuild any failed event's payload later.
  • A function's error rate shows which events were lost, so the destination needs no alarm.
  • A failure destination on the function also catches messages a polled queue keeps redelivering.
open as a page

How do a function platform's asynchronous retry count and maximum event age interact, and how would you set them for an unreliable downstream?

level: middleimportance: must knowfreq 58%

basics

~20 s

Retry count caps attempts; maximum event age caps time since acceptance, including time queued while throttled. Whichever runs out first ends the retries. Set the count from the handler's side effects and the age from how long the event stays useful.

open as a page

How do you choose a batch size and batching window for a queue-triggered function, trading message latency against invocation count?

level: middleimportance: should knowfreq 52%

basics

~20 s

A batch is sent when full, when the batching window ends, or at the payload cap. Bigger batches and longer windows cut invocations but add latency on quiet queues and enlarge the unit that fails together.

open as a page

After a weekend downstream outage failed thousands of asynchronous order-event invocations, how do you find every failed event and redrive it safely once the fix ships?

level: seniorimportance: should knowfreq 50%

basics

~20 s

Count three groups: events in the failure destination, events dropped before reaching it, and half-applied ones. Reconcile against the order store, test one replay, then redrive through the normal path at a controlled rate, parking superseded events.

open as a page

Why might a queue-triggered function process some messages twice with no invocation failing, and how do the function timeout and the queue's visibility timeout explain it?

level: seniorimportance: should knowfreq 48%

basics

~20 s

A received message is hidden for the queue's visibility timeout and deleted only after success. If processing outlasts that timeout and nothing renews it, the message reappears and a second invocation runs it while the first is still working.

open as a page

What event-delivery defaults should a platform team enforce for every asynchronously triggered function in an organisation, and what does each default cost the teams?

level: principalimportance: should knowfreq 32%

basics

~20 s

Mandate a failure destination with an owned alarm, enough retention and timeout ordering checks on every asynchronous function; derive retries and event age from declared side effects and freshness. The costs are alarm noise, storage, pipeline friction and exceptions.

open as a page