skip to content

What does Helicone-Retry-Enabled retry, and what does it cost the caller?

level: seniorimportance: should knowfreq 48%

answer

  1. retries live in the proxy, not the client
  2. transient statuses only, not your bad request
  3. four headers shape the backoff curve
  4. one client call, many upstream attempts
  5. stacking with the SDK multiplies attempts

basics

~20 s

Helicone-Retry-Enabled: true makes the proxy re-send a failed call to the provider on rate-limit 429s and 5xx errors, with exponential backoff. The retries happen inside one client request, so the caller sees a single much slower call and must widen its timeout.

solid answer

~50 s

Setting `Helicone-Retry-Enabled: true` turns on gateway-side retries. Helicone re-issues the upstream call when the provider returns a 429 or a 5xx, backing off exponentially between attempts; the shape of that backoff is tunable with `helicone-retry-num`, `helicone-retry-factor`, `helicone-retry-min-timeout` and `helicone-retry-max-timeout`. What it does not do is retry your mistakes — a 400 from a malformed request or an authentication failure is returned as-is, because retrying it would only fail again. The cost is paid in latency and in money. Every attempt is absorbed inside the single HTTP request your client made, so a call that used to take two seconds may now take thirty before it either succeeds or gives up, and any client-side timeout shorter than the total backoff will cut the sequence off mid-flight. Each attempt that reaches the model is billed, so a retry storm during a provider incident multiplies spend at exactly the moment you least want it.

code

python · 23 lines
python
from openai import OpenAI

client = OpenAI(
    base_url="https://oai.helicone.ai/v1",
    api_key="sk-your-openai-key",
    max_retries=0,  # let the gateway own retries, do not stack them
    timeout=45.0,   # must exceed the worst-case gateway backoff
    default_headers={
        "Helicone-Auth": "Bearer sk-your-helicone-key",
        "Helicone-Retry-Enabled": "true",
        "helicone-retry-num": "3",
        "helicone-retry-factor": "2",
        "helicone-retry-min-timeout": "1000",
        "helicone-retry-max-timeout": "10000",
    },
)

print(
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "ping"}],
    ).choices[0].message.content
)

go deeper

for a junior

Know that one header enables retries at the Helicone proxy and that they cover transient provider failures such as 429 and 5xx, not requests that were malformed to begin with.

for a middle

Explain that the retries happen inside a single client request, name the headers that shape the backoff, and work out the worst-case added latency before choosing an attempt count.

for a senior

Bring up the failure modes unprompted: client timeouts cutting the sequence short, retries stacking with the SDK's own, every attempt being billed, and a fleet-wide retry policy amplifying a provider incident.

for a principal

Decide where retry policy belongs across the organization — one gateway policy versus per-service client policy — and set the fleet-wide budget so a provider outage does not become a self-inflicted traffic multiplier.

## Where the retry happens The important structural fact is that Helicone retries on the server side of your request. Your application makes one HTTP call to the gateway; the gateway may make several calls to the provider before answering. This is convenient — you get retries without touching client code, in any language, including languages whose SDK has no retry support — and it is also the source of every gotcha. ## What is retried Helicone retries provider responses that are plausibly transient: rate-limit 429s and server-side 5xx errors. Client errors are not retried. A 400 because your JSON schema was invalid, a 401 because the provider key is wrong, a context-length error — these are deterministic, and re-sending them wastes time and produces the same failure. If your working theory is that enabling retries hardens the integration against all failures, the first malformed-request incident will correct you. ## Tuning the backoff Four headers shape the sequence: the number of attempts, the exponential factor applied between them, and a minimum and maximum wait. HTTP header names are case-insensitive, and Helicone's documentation writes these in lowercase — `helicone-retry-num`, `helicone-retry-factor`, `helicone-retry-min-timeout`, `helicone-retry-max-timeout`, with the timeouts in milliseconds. Multiply them out before you deploy. Three attempts with a factor of two starting at one second is a few seconds of added worst-case latency; five attempts starting at two seconds is most of a minute, which is longer than most HTTP clients, load balancers and browser fetches are willing to wait. ## The timeout interaction This is the trap worth raising unprompted. Your client's request timeout must exceed the worst-case total: the sum of all backoff waits plus the duration of every attempt. If it does not, the client abandons the connection while the gateway is still retrying, and you get the worst of both — the user sees a timeout, and the provider calls that were already in flight are still billed. The same applies at every hop in between: an ingress or service mesh with a thirty-second ceiling will cut a sixty-second retry sequence regardless of what your client is willing to wait. ## Cost amplification Every attempt that actually reaches the model consumes input tokens and is billed. A 5xx returned after the model has begun work still costs something, and a 429 costs nothing but adds latency. During a broad provider incident, a fleet with aggressive retries turns one failure into three or five, which is the classic retry-storm dynamic: the upstream is already degraded and your response is to send it more traffic. If you enable gateway retries fleet-wide, keep the attempt count small and the factor genuinely exponential rather than nearly flat. ## Double-retry stacking Most provider SDKs already retry internally by default. Turning on Helicone retries without turning the client's own retries off gives you the product of the two — three client attempts times three gateway attempts is nine calls to the provider for one logical request, with a worst-case latency nobody budgeted for. Pick one layer and disable the other. Gateway-side is the better choice when you have many services in many languages and want one consistent policy; client-side is better when the caller needs to make decisions between attempts, such as falling back to a different model. ## Idempotency and streaming A chat completion is usually safe to retry — it has no side effects at the provider — but the moment the call performs an action, retries become a correctness question rather than a reliability one. Be deliberate about whether every request behind a given base URL is safe to send twice. Streaming responses also complicate matters: once bytes have begun flowing to your client, a failure part-way through is not something a proxy can transparently paper over, so do not assume streaming calls enjoy the same protection as buffered ones. ## Retries and your own quota If you also run a `Helicone-RateLimit-Policy`, remember that a quota rejection is itself a 429. Configure with care so that a request rejected by your own policy is not then retried into the same wall by your own gateway; the useful mental model is that retries are for the provider's transient failures, and quota rejections are a decision your application should surface rather than fight.

  • Your service already uses the provider SDK's built-in retries. What should change when you enable Helicone's?
    Turn one of them off. Left stacked, the attempt counts multiply — three client attempts over three gateway attempts is nine provider calls for one logical request, with a worst-case latency nobody sized for. Keep gateway retries when you want one policy across many services and languages; keep client retries when the caller needs to act between attempts, such as switching model.
  • Would you enable gateway retries on a streaming endpoint?
    Cautiously, if at all. Once the first bytes have been sent downstream, a mid-stream failure cannot be transparently re-attempted without the client seeing a partial answer followed by a second one. Retries are cleanest on buffered request-response calls; for streaming, handle failure explicitly in the client so it can decide whether to restart the generation or surface what it already has.
  • How does a retry policy interact with your own Helicone rate-limit policy?
    Badly, if you are not deliberate. Your quota rejects with a 429, and 429 is exactly what the retry logic treats as transient, so a request refused by your own policy can be re-thrown at the same wall until the attempts run out. Retries are meant for the provider's transient failures; a quota rejection is a decision your application should surface, not retry through.

saying these in an interview costs you the question

  • Believes retries also cover 400 and authentication failures
  • Leaves the client timeout below the total backoff and blames the gateway
  • Stacks SDK retries on top of gateway retries without noticing
  • Assumes retried attempts are not billed by the provider
  • Enables aggressive retries fleet-wide and worsens a provider incident

context

resolving links…