A Lambda function is triggered by s3:ObjectCreated:* on a bucket and writes its processed output back into that same bucket. What goes wrong, and how do you design around it?

## The mechanism An S3 notification does not care who wrote the object. If the function's trigger is `s3:ObjectCreated:*` on the whole bucket and the function does a `PutObject` back into that bucket, the put produces exactly the event the trigger is watching for. Invocation *n* creates the object that causes invocation *n+1*. If each invocation writes one object the growth is linear and merely expensive; if it writes several — one thumbnail per size, say — the growth is exponential and the meter moves very fast. ``` upload -> event -> lambda -> writes output/ -> event -> lambda -> writes output/ -> ... ``` ## What it actually costs Four separate meters run at once, and the third is the one that turns an expensive mistake into an incident: - **Lambda duration and requests** for every invocation in the chain. - **S3 request charges** for every PUT and every GET the function makes, which on a high-rate loop can exceed the compute cost. - **Account-wide concurrency.** Lambda concurrency is a shared, regional pool. A runaway that climbs to the account limit throttles *other* functions — the API-backing function, the queue consumer — and a storage mistake becomes a site outage. - **Downstream amplification.** Whatever the function calls (a database, a third-party API) receives the same runaway rate. ## Fixing it structurally Ranked from most to least robust: 1. **A separate output bucket.** The input bucket has the notification; the output bucket has none. There is no filter to get wrong and no way for a later change to a key convention to reintroduce the loop. This is the default worth arguing for. 2. **Disjoint prefixes with a filter on the trigger.** Trigger on `prefix: incoming/`, write to `processed/`. Correct, cheap, and the standard single-bucket pattern — but it depends on the filter staying in place. If someone later broadens the notification to the whole bucket, or a new code path writes back into `incoming/`, the loop returns. 3. **Disjoint suffixes.** Trigger on `suffix: .raw.json` and write `.out.json`. Same tradeoff as prefixes, and more fragile because extensions are easy to change. 4. **A guard inside the handler.** Check the key against the output convention and return early. This still costs one invocation per written object — it bounds the blast radius but does not remove the recursion, and it is the weakest of the four because the fence lives in code rather than in configuration. A useful discipline for reviewing any S3-triggered function: state, in one sentence, why the objects this function writes cannot match its own trigger. If the sentence is hard to write, the fence is not real. ## Bounding the damage before it happens Even with a correct fence, treat unbounded invocation as a failure mode worth capping: - **Reserved concurrency** on the function sets a hard ceiling on how many copies can run at once. It caps the loop's rate and, just as importantly, protects the rest of the account's functions from being starved by this one. - **A CloudWatch alarm on the function's invocation count** over a short period catches the runaway in minutes rather than at the end of the billing cycle. Duration and error-rate alarms will not: in a loop every invocation succeeds. ## Stopping one that is already running Speed matters more than elegance. Two effective moves: - **Set the function's reserved concurrency to zero.** This throttles every invocation immediately and is fully reversible. It is the fastest kill switch, at the cost of stopping legitimate work too. - **Remove the notification configuration from the bucket.** This stops new events at the source. Anything already in flight still runs. Deleting the objects the loop created is cleanup, not containment — do it after the trigger is severed, or you are deleting into a running loop. And when you re-enable, remember that S3 does not backfill: the legitimate uploads that arrived while the trigger was off will not be replayed, so plan a batch pass over them. ## The interview version Name the loop, name the account-wide concurrency blast radius (this is the detail that separates a senior answer from a textbook one), give the separate-bucket fix as the structural default, and mention reserved concurrency plus an invocation-count alarm as the cap. Then say how you would stop one at three in the morning.