You are designing a workflow backed by AWS Lambda and must decide whether the caller retries a failed synchronous invoke, or the work is handed to Lambda's asynchronous invocation path so Lambda retries it. How do you make that call, and what does each choice cost you?

## Reframe the question "Who retries" sounds like a configuration detail. It is actually a decision about where the authority over failure lives, and it determines the contract your system offers: does the caller get an answer, or a promise? ## The deciding questions **Is anyone waiting on the result?** If a user or an upstream service needs the outcome to proceed, you are synchronous by necessity. If the work only has to happen eventually, asynchronous is available and usually better. **What is the latency budget?** Lambda's asynchronous retries are spaced in minutes. If the work must complete inside a few seconds, that policy is unusable regardless of how appealing the durability is. **What happens if it never completes?** Work with financial or legal consequence needs durable custody and a capture path. Best-effort work — a cache warm, an enrichment — can be dropped. **Is the handler idempotent?** If it is not, every retry mechanism is dangerous and the design work is idempotency, not retry placement. ## Caller-side retry: control, and a loaded gun Retrying a synchronous invoke keeps everything visible. The caller sees the error immediately, applies its own policy, can distinguish a retryable dependency failure from a permanent validation error, and can degrade gracefully — serve stale data, queue for later, tell the user honestly. The danger is amplification. Each retry is another concurrent execution. Three attempts per caller against a slow dependency means three times the in-flight requests and three times the concurrency consumed, at precisely the moment the system is least able to afford it — and Lambda's concurrency is a shared account-level resource, so an amplifying caller starves functions that have nothing to do with the incident. That is the metastable failure pattern: the retries keep the system down after the original trigger has passed. So caller-side retry is only safe with the full discipline: a small attempt cap, exponential backoff **with jitter**, retrying only errors that can plausibly succeed on a second try, a circuit breaker that stops trying when the dependency is clearly down, and a deadline that ends the whole thing. Modern AWS SDKs give you part of this — configurable retry modes and a maximum attempt count — but their policy covers throttling and service errors, not your application's semantics, so you still own the decision about *your* failures. ## Lambda-side asynchronous retry: durability, and a policy you do not control Invoking asynchronously moves custody to AWS. The caller gets a 202 in milliseconds and is finished. Lambda holds the event durably, retries handler failures a bounded number of times, and — importantly — keeps re-attempting throttled invocations until the event's age limit, turning a concurrency shortage into queue depth instead of a cascade of errors. That backpressure property is the strongest argument for this path. The costs, stated honestly: - **The policy is coarse and nearly untunable.** At most two retries, minute-scale delays you do not choose, no per-error-class behaviour, no jitter control. - **The caller is blind.** No result, no error, ever. Failure observability has to be built: an on-failure destination so payloads survive, alarms on `Errors` and on delivery failures to the target, and attention to `AsyncEventAge` as the backlog signal. - **Ordering and timing are unspecified.** Events may arrive out of order and may be delivered more than once. - **A poison event is expensive.** It burns three invocations before dying, and a flood of them triples volume and cost. ## The third option, and why to name it When neither policy fits — you need ten attempts, or hours of backoff, or explicit per-message failure handling — the honest answer is that you want an explicit queue between the caller and the function, so the retry policy becomes yours to write rather than AWS's to impose. Say so in an interview: recognising that the built-in asynchronous policy is a convenience with fixed parameters, and knowing when your requirements have outgrown it, is exactly the judgment being probed. ## The rule about layering Retries compose multiplicatively. A caller with three attempts, invoking asynchronously into a path with three attempts, against a handler that itself retries a dependency three times, produces up to twenty-seven executions of one logical request. Pick **one** layer to own retry, make every other layer fail fast and propagate, and write that down where the next team can find it. Retry budgets that are implicit in three separate codebases are how a small dependency wobble becomes an all-hands incident. ## What a strong answer sounds like Start from who is waiting; pick the model that matches; state the cost you are accepting; name the single layer that owns retry; require idempotency in the handler regardless; and describe how failures become visible in each case — a returned error for the synchronous path, a destination plus alarms for the asynchronous one.