A function that writes its result into the same storage bucket that triggers it ran up a large bill overnight; what happened, and what stops it recurring?
answer
- the output is the trigger
- each run starts the next
- retries multiply across layers
- cap concurrency, alarm on invocations
- billing data arrives late
basics
~20 sEach result written to the bucket fired the trigger again, so the function looped and scaled out until a limit or a person stopped it. Write output elsewhere, cap concurrency and alarm on invocation rate; billing alerts lag by hours.
solid answer
~40 sThe function's own write was an event the trigger listens for, so every run started another: a recursive loop the platform scaled out to meet, paying compute plus storage requests and log ingestion on every round. First I stop it: set the function's concurrency cap to zero or disable the trigger, and purge the backlog so retries do not restart it. To prevent it, write output to a separate bucket or prefix and filter the trigger to the input prefix, have the handler skip objects it wrote, and rely on the platform's loop detection only as a backstop. A per-function concurrency cap bounds the rate of spend, and an alarm on invocation rate catches it within minutes, because a budget alert works from billing data that arrives hours later.
code
pseudocode · 11 lineshandler(event):
key = event.object.key
# guard 1: never process our own output
if key starts with "processed/":
return
# guard 2: skip objects this function wrote (marker in metadata)
if event.object.metadata["written-by"] == FUNCTION_NAME:
return
result = transform(read_object(key))
write_object("processed/" + key, result,
metadata = { "written-by": FUNCTION_NAME })go deeper
Recall that a function writing into the place that triggers it will trigger itself, and that the fix is to write the output somewhere the trigger does not watch.
Explain how retries at the client, the library and the platform multiply, and why a timed-out attempt is billed for the whole time limit on every try.
Walk through the incident: kill switch first, backlog decision, root-cause fix, then layered guards and an invocation-rate alarm that acts within minutes.
Argue for fleet defaults: bounded retries at one layer, a concurrency cap on every function, and invocation alarms wired to automation, accepting some throttling to bound worst-case spend.
## How a function bill runs away A function platform scales out to meet whatever invokes it, and it bills every invocation. That is the model working as designed, which is why a **cost blow-up** on functions rarely announces itself as an outage: nothing is down, the platform is simply doing more and more work. Two shapes cause most surprise bills: a **recursive loop**, where a function's own output triggers it again, and **retry amplification**, where several layers each retry the same failure. ## Recursive loops: the output is the trigger The classic case: a function is triggered when an object lands in a storage bucket, processes it, and writes the result back into the same bucket. The write fires the trigger, the new invocation writes again, and each round starts another. Because every invocation can produce one or more outputs, the loop grows until it reaches a **per-function concurrency cap**, a regional concurrency quota, or someone notices. Other shapes behave the same way: - a function that publishes to the topic or queue it consumes from; - a function that invokes itself to continue work and has no stop condition that is guaranteed to fire; - two functions that trigger each other through a bucket, a table change stream or a queue; - a failure handler that re-publishes the failed event to the original source. Every round pays more than compute: storage write and read requests, log ingestion for each invocation, queue or topic requests, and egress if anything leaves the region. ## Retries as a multiplier Retries are configured at several layers, and they **multiply** rather than add: 1. The client retries a failed call. 2. The function's client library retries its call to a dependency. 3. The platform's asynchronous invocation path, or a queue's redelivery, runs the whole invocation again. If each of three layers makes up to four attempts, one original request can turn into dozens of calls on the failing dependency, each one billed. A **timeout** makes it worse: an attempt that runs until the function's time limit is billed for the whole limit, and every retry bills it again. A dependency outage therefore raises the bill exactly when nothing useful is being done. ## Stopping it now 1. **Kill switch:** set the function's concurrency cap to zero, or disable its trigger. Either stops new invocations in minutes. 2. Confirm invocation count has dropped to zero on the platform's metrics. 3. Decide what to do with the backlog: queued and asynchronous events may be retried once the cap is lifted, so purge or park them before re-enabling. 4. Fix the cause, then lift the cap gradually while watching invocations per input. ## Guards that stop it recurring | Guard | What it bounds | Limitation | |---|---|---| | Separate output location | breaks the storage loop at the source | relies on every writer following the convention | | Trigger filter on input prefix | output never matches the trigger | only where the trigger supports prefix or suffix filters | | Marker on own output | handler skips objects it wrote | costs one wasted invocation per output | | Platform loop detection | stops a loop after a bounded number of round trips | covers only the paths the platform tracks | | Per-function concurrency cap | caps the rate of spend | the loop keeps running at the capped rate | | Invocation-rate alarm | detects the loop within minutes | someone, or automation, must act on it | Some platforms detect a loop that runs through their own queue, topic or storage services and stop it; treat that as a backstop, not the design. Bound retries at one layer, preferably the outermost one, with backoff, and set function timeouts close to observed duration plus a margin rather than at the platform maximum. ## Why the budget alert came too late Billing data is aggregated and published with a delay that is often measured in hours, so a **budget alert** fires long after a fast loop has spent the money, and by default it notifies rather than stopping anything. The signal that keeps up is the platform's **invocation and concurrency metrics**, which arrive within minutes. An alarm on invocation rate per function, or on concurrency jumping well above its usual ceiling, catches a loop early; wiring that alarm to apply the kill switch automatically turns a large bill into a small one. Keep budget alerts and cost-anomaly detection as the second net for slower leaks.
- Why does a per-function concurrency cap bound the cost of a loop without stopping it?The cap limits how many copies run at once, so spend per hour has a ceiling, but the loop keeps going at that rate until its cause is removed. Events that could not run may be retried later, so when the cap is lifted, the loop restarts unless the backlog is purged or the trigger is fixed first.
- Why is a timeout an expensive failure on top of a retry?An attempt that runs until the function's time limit is billed for the whole limit, and each retry bills it again while doing no useful work. Setting the timeout near observed duration plus a margin, and failing fast on a dead dependency, keeps a dependency outage from becoming a large bill.
- How do you prove the loop is fixed before lifting the kill switch?Replay a few input objects in a non-production copy and check that each produces exactly one invocation and one output. Then lift the cap in steps in production while watching invocations per input object and concurrency against their usual levels.
- A budget alert is enough; it will halt the loop when it fires.
- The platform's loop detection catches every recursive path, so code needs no guard.
- Retries at each layer add a few attempts; they never multiply.
- A loop only raises compute; storage, request and log charges stay flat.
- Raising memory limits a loop's damage, because each looped run finishes sooner.