skip to content

Why might a queue-triggered function process some messages twice with no invocation failing, and how do the function timeout and the queue's visibility timeout explain it?

level: seniorimportance: should knowfreq 48%

answer

  1. hidden, not deleted, while processing
  2. deleted only after success
  3. hiding ends before the work does
  4. receive count climbs on good messages
  5. caller, function, then queue

basics

~20 s

A received message is hidden for the queue's visibility timeout and deleted only after success. If processing outlasts that timeout and nothing renews it, the message reappears and a second invocation runs it while the first is still working.

solid answer

~50 s

When the poller receives a message, the queue hides it for the **visibility timeout** (a lease, on some systems) and deletes it only after the function returns success. If nothing renews that lease and the function's timeout, plus the time the message waited in a batching window, is longer than the visibility timeout, a slow invocation can still be running when the message becomes visible again. The poller hands it to a second instance and both process it: a duplicate with no error anywhere. The receive count also climbs, so a healthy message can even land in the dead-letter queue. I confirm it by finding one message ID handled by two overlapping invocations, then fix the ordering: downstream call timeout below the function timeout, and the function timeout plus batching window well below the visibility timeout. Handlers still need to be idempotent, since redelivery can happen for other reasons.

go deeper

for a junior

Remember that a polled message is hidden while it is processed and only deleted after success, so a short hiding period lets it reappear.

for a middle

Explain the race step by step: visibility expires, the message reappears, a second invocation runs it, and the receive count climbs.

for a senior

Show the diagnosis, overlapping invocations for one message ID, and the nesting of downstream, function and visibility timeouts with the batching window included.

for a principal

Make timeout ordering a checked rule in deployment templates, so no team ships a function timeout that outruns its queue's visibility timeout.

## How a polled queue hands out a message A queue that a function platform polls does not delete a message when it is read. It **hides** it. When the poller receives a message, the queue makes it invisible to other readers for a period called the **visibility timeout** (pull-based systems call the same idea an **acknowledgement deadline** or a **lease**). The message is deleted only when the reader acknowledges it, and the platform's poller acknowledges a batch only after the function returns success. That design makes failure safe: if the reader crashes, the hiding period ends, the message reappears and someone else processes it. It also means the queue has no idea whether a reader is still working. It only knows the clock. ## The race that produces a duplicate Duplicates without any failure come from three timeouts that disagree: - **Downstream call timeout** - how long the handler waits for a database or an API. - **Function timeout** - how long the platform lets one invocation run before killing it. - **Visibility timeout** - how long the queue keeps a received message hidden. If the function can legitimately run longer than the message stays hidden, this happens on a slow day: 1. The poller receives a batch; the visibility clock starts. If the batch then waits in a batching window, that waiting also uses up visibility time. 2. The function starts and a slow downstream stretches it toward its own timeout. 3. The visibility timeout ends while the function is still working. The message reappears. 4. The poller receives it again and invokes a second instance. 5. Both invocations finish successfully. The customer gets two confirmations, or the downstream receives two writes. No invocation errored and no retry was configured. The duplicate came from the clocks alone. ## The side effects you see - **Duplicates that correlate with slow periods**, not with errors or deployments. - A **receive count above one** on messages that eventually succeeded. - **Healthy messages in the dead-letter queue**: the queue counts each receive, so a message redelivered several times by the race reaches the maximum receive count and is moved aside although it never failed. - **Invocation duration close to the function timeout** in the same window. ## Diagnosing it 1. Compare the configuration: function timeout plus batching window against the queue's visibility timeout. 2. Search logs for a **message ID** handled by two invocations whose start and end times overlap. Overlap is the fingerprint; a crash-and-redeliver duplicate would not overlap. 3. Rule out the producer: a producer that retries a send creates two messages with **different IDs**. Same ID means the queue redelivered. 4. Check the receive count on duplicated messages and duration percentiles in the same period. ## Fixing the ordering The timeouts should nest, each one inside the next: | Timeout | Set it relative to | Why | |---|---|---| | Downstream call timeout | Below the function timeout | The handler fails cleanly and records the error instead of being killed mid-write | | Function timeout | Close to the real worst-case duration, with headroom | A runaway invocation is stopped before it overlaps redelivery | | Visibility timeout | Well above function timeout plus batching window | A message stays hidden until its invocation has certainly ended | Where a platform publishes a recommended margin between function timeout and visibility timeout, it is typically a multiple, not a small addition, because throttled or retried invocations of the same batch also consume visibility time. Read your platform's guidance rather than inventing a margin. Other fixes worth weighing: - **Shorten the function timeout** to the real need instead of leaving it at the maximum "to be safe". A maximal function timeout forces a maximal visibility timeout, which slows recovery after a genuine crash. - **Smaller batches** reduce the time one invocation needs. - Some platforms and client libraries **renew the lease** while work is in progress, up to a configured limit; where that is on, it removes the race within that limit, at the cost of a longer wait after a real crash. ## What alignment does not fix Aligned timeouts remove one cause of duplicates. They do not make delivery exactly-once: crashes after a side effect and before acknowledgement, and producer retries, still deliver twice. Handlers still need idempotent processing; that pattern is covered with idempotent consumers. The operator's job here is to stop the configuration from manufacturing duplicates on every slow day.

  • Why can a message that never failed end up in the queue's dead-letter queue in this situation?
    The queue counts each time the message is received, not each failure. When the visibility timeout keeps expiring during slow invocations, the same message is received repeatedly. Once that count reaches the maximum receive count, the queue moves it to the dead-letter queue, even if every invocation eventually succeeded.
  • Why not just set the function timeout and the visibility timeout both to the maximum?
    A maximal function timeout lets a hung invocation hold a message and an instance for the longest possible time, and a maximal visibility timeout delays redelivery after a genuine crash by the same amount. Set the function timeout from measured worst-case duration, and the visibility timeout from that with the platform's recommended margin.

  • A message is deleted from the queue as soon as the function receives it.
  • Every queue extends visibility automatically while the function is still running.
  • Duplicates without errors must come from the producer sending twice.
  • The function is killed when the queue's visibility timeout expires.
  • Matching the visibility timeout exactly to the function timeout is enough.