A function's traces and metrics are missing for a fraction of invocations, though its logs show every request succeeded; how do you diagnose and fix it?
answer
- who sends the data, and when
- a background timer in a frozen sandbox
- worse on quiet functions
- flush before return, or hand it off
basics
~20 sTelemetry libraries buffer data and export it on a background timer, which stops when the sandbox is frozen after the response, so buffered spans arrive late or are lost. Flush before returning, or export through a local extension or log lines.
solid answer
~50 sLogs usually survive because the platform captures standard output as it is written, while a tracing or metrics library **buffers** data in process memory and exports it in batches on a **background timer**. On a platform that freezes the sandbox after the response, that timer stops: buffered spans sit in a frozen sandbox, arrive late if a later invocation thaws it, and are lost if the sandbox is reclaimed first. Confirm it by comparing the platform's invocation count with spans received per function and hour (the gap is worst on low-traffic functions), checking for spans that arrive long after their timestamps, and ruling out sampling. Fixes: force a flush before returning, which adds export latency to every response; export through a local extension process with its own lifecycle; or write telemetry as structured log lines. A **shutdown hook** is a last chance only: its grace window is short and some terminations skip it.
code
pseudocode · 11 lineshandler(event):
span = tracer.startSpan("process-order")
try:
result = processOrder(event)
return result
finally:
span.end()
await tracer.forceFlush(timeout = shortTimeout) // export before the sandbox freezes
onShutdown():
tracer.forceFlush(timeout = withinGraceWindow) // safety net, not delivered on every terminationgo deeper
Recall the cause: telemetry is buffered and sent by a background timer, and that timer stops when the sandbox is frozen after the response.
Explain why logs survive and spans do not, and what each fix (flush, extension, log lines, shutdown hook) trades.
Show the diagnosis you would run: spans per invocation by traffic level, sampling ruled out, arrival delay, losses clustered at the end of sandbox lifetimes.
Frame it as a fleet default: one supported telemetry path for every function, its latency budget, and how teams prove their data is complete.
## Why telemetry is fragile in a function sandbox Most tracing and metrics libraries are designed for long-running servers. They **buffer** spans and data points in process memory and send them in batches from a **background timer**, so that the request path never waits on the telemetry backend. On a function platform that freezes the sandbox after the handler returns (the normal behaviour on microVM-based platforms), that design breaks: - The timer does not fire while the sandbox is frozen, so the last batch of each busy period stays in memory. - If a later invocation thaws the same sandbox, the batch is exported late, with timestamps long before it arrives. - If the platform reclaims the idle sandbox first, the batch is lost with it. Nothing records the loss, because the code that would report it never ran. **Logs** usually survive the same lifecycle because the platform captures standard output from outside the handler as lines are written. That holds only while the logging library writes straight to standard output; a logger that buffers in process before writing has the same problem as the tracer. ## How the loss shows up - Invocation counts on the platform's own metrics exceed the request spans in the tracing backend, and the gap is **largest on low-traffic functions**, where most invocations are the last one before a sandbox goes idle. - Traces end abruptly at the function's boundary, with downstream spans present but the function's own span missing. - Spans arrive minutes or hours after their timestamps, and some dashboards show them in the wrong window. - The gap shrinks during load tests, because constant traffic keeps thawing sandboxes, and comes back at night. ## Diagnosing it 1. **Compare counts.** For each function and hour, divide spans received by invocations reported by the platform. A ratio that falls as traffic falls points at the freeze, not at the backend. 2. **Rule out sampling.** Confirm the sampler's configured rate; a sampler dropping data on purpose is not loss. 3. **Measure arrival delay.** Plot receipt time minus span timestamp. A long tail of late arrivals shows thawed sandboxes exporting old batches. 4. **Read the exporter's own counters**, where it publishes them: spans dropped, export failures, queue length at shutdown. 5. **Correlate with sandbox lifetimes.** Tag each span with a sandbox identifier generated at start-up; missing spans cluster on the last invocations before a sandbox disappears. ## Fixes and what each costs | Approach | How it works | Cost | |---|---|---| | Flush before return | The handler calls the exporter's force-flush and awaits it | Export latency added to every response | | Local extension process | The handler hands telemetry to an extension or sidecar the platform runs beside it, which ships it on its own lifecycle | Extra memory, and the mechanism is platform-specific | | Telemetry as log lines | Spans and metrics are written as structured lines to standard output, and a pipeline turns them back into telemetry | Log ingestion cost and a parsing step | | Shutdown hook flush | The runtime flushes when the platform signals teardown | Last resort only: short grace window, not delivered in every termination | The usual answer is a combination: a flush or extension for the request's own telemetry, plus a shutdown hook as a safety net for anything still buffered. ## What a shutdown hook does and does not promise A **shutdown hook** is code the runtime runs when the platform tears a sandbox down, often triggered by a termination signal or a lifecycle event sent to an extension. It gives a **short grace window**, typically measured in seconds, to flush buffers and close connections. Operators should treat it as best-effort: - It is not delivered when the process crashes, and on some platforms not when an invocation hits the hard time limit and the sandbox is reset. - It runs once per sandbox, not once per invocation, so it cannot replace a per-request flush. - Work in it must be short and must not depend on network calls that could hang past the window. - Whether it runs at all can depend on the platform's configuration, such as whether an extension is registered.
- Why does the loss shrink during a load test and come back at night?Under constant traffic every sandbox is invoked again within moments, so its frozen batch is thawed and exported almost at once, and few sandboxes are reclaimed while holding data. At night most invocations are the last one before a long idle period, so their batches are either exported very late or lost when the platform reclaims the sandbox.
- What does a forced flush before return cost, and how do you bound it?Every response waits for the export round trip, so latency and billed duration rise, and a slow telemetry backend now slows the function. Bound it with a short flush timeout, export to a nearby collector or local extension rather than a distant backend, and accept dropping a batch over failing the request.
- Why can a shutdown hook not be the only flush?It runs once per sandbox at teardown, not after each invocation, so data waits in memory until then. Its grace window is short, and it is not delivered when the process crashes and, on some platforms, when an invocation times out. It is a safety net for leftovers, not the delivery path.
- If the logs are complete, the telemetry pipeline must be complete too.
- Missing spans mean the tracing backend is overloaded or rate-limiting the function.
- Lowering the exporter's batch interval fixes it, because the timer fires more often.
- A shutdown hook runs after every invocation, so it can flush each request's data.
- The loss is random, so it is the same on busy and quiet functions.