skip to content

After quiet periods, a function's first call to a downstream service fails with a connection-reset error and succeeds on retry; how does the sandbox lifecycle cause this?

level: seniorimportance: should knowfreq 35%

answer

  1. the pool outlives the invocation
  2. nobody answers while frozen
  3. idle timeouts on the far side
  4. pool idle timeout shorter than the path's

basics

~20 s

A pooled connection kept from an earlier invocation sits idle while the sandbox is frozen. The server, load balancer or NAT closes it at its idle timeout, the frozen client never notices, and the first write after the thaw fails.

solid answer

~50 s

Connections opened during one invocation stay in the client's pool for the next, which is the point of reusing a warm sandbox. But while a microVM-based sandbox is frozen, the whole guest is paused, so it sends no keepalives and processes no close from the far side. Meanwhile the server, a load balancer or a NAT gateway closes or forgets the idle connection at its own **idle timeout**. On the thaw, the pool hands out a socket the client still believes is open; the write draws a reset, or simply hangs until a timeout if a middlebox silently dropped the flow. The retry opens a fresh connection and works. The fix is to set the client pool's **idle timeout shorter than the shortest idle timeout on the path**, discard connections that have been idle too long when the sandbox resumes, and retry once on a reset, but only for idempotent requests.

code

yaml · 11 lines
yaml
# illustrative client pool settings for a function behind a NAT gateway
# whose idle timeout is the shortest on the path
httpClientPool:
  idleTimeout: shorterThanShortestHopIdleTimeout
  validateIdleAgeOnCheckout: true
  connectTimeout: short
  readTimeout: belowFunctionTimeout
  retryOnConnectionReset:
    enabled: true
    onlyIdempotentMethods: true
    maxAttempts: 2

go deeper

for a junior

Recall that pooled connections survive between invocations, and that the far side can close them while the sandbox is frozen.

for a middle

Explain the idle timeouts on each hop and why a frozen sandbox cannot send keepalives or notice a close.

for a senior

Show the diagnosis (failures only after a gap, clustered above one hop's idle timeout) and the fix: pool idle timeout, idle-age checks and idempotent-only retries.

for a principal

Frame it as a shared client default for every function in the organisation, kept in step with the network team's idle timeouts as infrastructure changes.

## The connection outlives the invocation Reusing a warm sandbox is valuable precisely because state survives between invocations: clients, caches and, above all, **pooled network connections**. A pooled connection saves a TCP and TLS handshake on every call. The catch is that the connection's other end does not know about the sandbox lifecycle, and it keeps its own clock. Every hop on the path has an **idle timeout**: the downstream server closes connections idle too long, a load balancer drops idle flows, and a NAT gateway forgets address mappings it has not seen traffic for. A long-running server keeps its connections fresh by using them, by sending keepalives, or by noticing the close when it arrives. A frozen sandbox can do none of these. ## What happens during the freeze 1. Invocation A opens a connection to a downstream service through a load balancer and a NAT gateway, uses it, and returns. The client's pool keeps the connection. 2. The platform freezes the sandbox. On a microVM-based platform the whole guest is paused, kernel included, so neither application pings nor kernel keepalives are sent, and a close packet from the far side is not processed. 3. A quiet period passes that is longer than some hop's idle timeout. That hop closes the connection, or silently drops its record of it. 4. Invocation B thaws the sandbox. The pool hands out the old connection, which the client still believes is open, and the request is written to it. 5. The outcome depends on what the hop did: - if the server or load balancer closed the connection, the write draws a **reset** and fails fast; - if a NAT gateway or firewall silently forgot the flow, packets vanish and the call **hangs** until the client's own timeout fires, which is worse. The retry opens a new connection, so it succeeds, and the incident looks like a transient network blip. ## Why it looks intermittent - It happens only on the **first** call after a gap, and only when the gap exceeds the shortest idle timeout on the path. - Busy functions rarely see it, because their sandboxes are thawed again within moments. - It moves when infrastructure changes: adding a NAT gateway or a load balancer with a shorter idle timeout makes it appear on functions that never showed it. - Whether keepalives help depends on whether the kernel handling the socket is paused with the sandbox. On a microVM platform it is; on some container-instance platforms the kernel keeps running while the process is throttled, so kernel keepalives may still be sent. ## Fixes and their caveats | Fix | What it does | Caveat | |---|---|---| | Pool idle timeout shorter than the shortest idle timeout on the path | The client discards a connection before any hop can | Needs the real timeouts of every hop, which change with infrastructure | | Check idle age on use | Before handing out a connection, discard it if it has been idle longer than a threshold | Costs one extra handshake after every quiet period, which is the price of correctness | | Retry once on a connection reset | Turns the failure into a short delay | Safe only for idempotent requests | | Short connect and read timeouts | Bounds the silent-drop case so it fails fast instead of hanging | A too-short timeout fails healthy slow calls | Relying on TCP keepalives alone does not work on a platform that pauses the kernel, because no keepalive is sent while frozen. ## Retries and idempotency A reset can also arrive after the server has received and processed the request, for example when the connection breaks while the response is in flight. So a blind retry is safe for reads and for writes carrying an **idempotency key**, and unsafe for a non-idempotent call such as a payment without one. The durable fix is the pool configuration; the retry is a safety net for what slips through. ## Confirming the diagnosis - Tag each outbound error with the time since the sandbox's previous invocation; failures cluster above one threshold. - Compare that threshold with the idle timeouts configured on the load balancer, the NAT gateway and the server. - Check that the failing call is the first one on that connection after the gap.

  • Why can the same bug show up as a hang instead of a reset?
    A NAT gateway or stateful firewall that hits its idle timeout often forgets the flow without telling either side. When the thawed client writes, its packets are dropped or answered by nobody, so nothing fails fast; the call waits until the client's read or connect timeout. That is why short client timeouts belong with the pool fix.
  • Does enabling TCP keepalives fix it?
    Not on a microVM-based platform: the guest kernel is paused with the sandbox, so no keepalive is sent during the freeze, exactly when it would be needed. Keepalives help only where the kernel handling the socket keeps running while the function is idle. The pool's idle timeout works on every model.

  • The platform closes every open socket when it freezes a sandbox, so pools never go stale.
  • Connection pooling is unsafe in functions and should be disabled.
  • TCP keepalives always keep the connection alive through the freeze.
  • A connection reset is always safe to retry, whatever the request does.
  • The error is a downstream outage, because the retry succeeded a moment later.