How does the Langfuse SDK cache prompts fetched with get_prompt, and what is the default TTL?

## The problem the cache solves Server-side prompts turn a string literal into a network read. If every request called Langfuse for the prompt text, you would have added a third-party dependency to the hot path of your application, plus tens of milliseconds of latency per call and a large volume of read traffic. The SDK's client-side cache exists so that in the steady state, fetching a prompt is a dictionary lookup. ## How it behaves - **Scope**: in-process and in-memory. Each Python process holds its own cache; nothing is shared between replicas, and the cache dies with the process. - **Key**: the prompt name together with what you asked for — a specific `version=`, a specific `label=`, or the default `production` resolution. Fetching the same prompt under two labels caches two entries. - **Default TTL**: 60 seconds. - **Expiry behaviour**: expiry does not evict. When a cached entry is stale, `get_prompt` returns the stale value immediately and kicks off a background refresh, so the caller does not block. The refreshed value is served by subsequent calls. - **Cold start**: the first `get_prompt` for a given key in a process has nothing to serve, so it does block on the network. That design is deliberately biased toward availability and latency over freshness: a slightly old prompt is almost always better than a slow or failed request. ## Tuning the TTL `cache_ttl_seconds` is a per-call argument: - **Default (60s)** — fine for most services. A promotion is live everywhere inside about a minute. - **Longer (e.g. 300s)** — fewer reads, more resilience to a Langfuse blip, slower propagation. Reasonable for a stable prompt on a very high-QPS path. - **Shorter** — faster propagation at the cost of more requests; rarely worth it, since the background refresh already keeps the blocking cost at zero. - **`cache_ttl_seconds=0`** — caching off, network round trip on every call. Correct for a one-shot script, a notebook, or while iterating on a prompt and wanting each run to see the newest text. Wrong on a request path: you have re-introduced the dependency and the latency you were avoiding. ## Propagation is per-process, not fleet-wide This is the consequence engineers most often miss. Moving the `production` label is a single write on the Langfuse side, but each of your replicas only notices when its own entry expires and refreshes. With the default TTL and, say, 20 replicas, the fleet converges within roughly a minute — but during that minute **both versions are serving traffic**. Practical implications: - A promotion is a gradual rollout with an uncontrolled schedule, not an atomic switch. That is usually harmless and occasionally not — for instance if the prompt change is paired with a code change that expects a new variable. - A rollback has the same lag. In an incident, "I moved the label back" does not mean the bad prompt has stopped serving; it means it will stop within about a TTL. - To reason about which version served a given request, you need the version recorded on the generation itself. The label tells you the current pointer, not what was used a minute ago. ## Warm-up and long-lived processes Because the cost lands on the first call, a natural pattern is to fetch the prompts a service uses at startup, so the first real user request finds a warm cache. This also surfaces a missing prompt or a missing `production` label at boot rather than on the first customer request. In serverless environments where processes are short-lived and frequently cold, expect the blocking first fetch far more often — a case for a longer TTL, a warm-up in the init phase, and definitely for supplying a fallback. ## Interaction with failure The cache is also the first line of defence when Langfuse is unreachable: a process that already holds a copy keeps serving it rather than failing. The TTL you pick therefore doubles as a freshness-versus-resilience dial, not just a request-volume dial.