Why can two identical Helicone cache hits return different completions?

## The default: one response per key Helicone's cache maps an exact request to a stored response. Enable it, send the same request twice, and the second call is answered from storage with the identical text. For deterministic work — classification, extraction, translation — that is exactly right, and the fact that the answer never varies is a feature. ## Bucketing For open-ended generation it can read as a defect. If a hundred users ask an assistant the same broad question and every one of them receives a byte-identical answer, the product feels canned in a way that a fresh generation would not. `Helicone-Cache-Bucket-Max-Size: N` addresses this by storing up to N distinct responses under one key. The first N matching requests miss and are generated normally, filling the bucket. Once it is full, subsequent hits return one of the stored responses selected at random. So the header buys variation at a controlled price: instead of one generation per key you pay N, and every request beyond that is free. Helicone reports which slot answered on the response alongside the usual HIT or MISS indication, so you can confirm bucketing is behaving during rollout. ## The determinism you just gave up This is the part worth being explicit about in an interview, because it is where bucketing actually bites. Caching normally makes a system more repeatable; bucketing makes a cached system less repeatable than an uncached one is per key, because the variation is now sampled from a fixed pool at hit time and you have no control over which member you get. Concretely, this breaks: - Golden-output tests. A suite that asserts an exact response, or diffs against an approved snapshot, will pass or fail depending on which bucket slot was served. In CI that surfaces as a flaky test with no code change behind it, which is corrosive — teams start re-running until green and stop believing the suite. - Idempotency schemes that hash the response. If a downstream system dedupes on a content hash, the same logical request now produces several distinct hashes. - Debugging. "Reproduce it locally" stops working when the reproduction depends on which of three stored answers you happen to draw. The mitigation is boring and effective: do not bucket the requests your automated checks run against. Bucketing is a request-level header, so a test harness can simply send `Helicone-Cache-Bucket-Max-Size: 1`, or omit caching entirely, while production traffic buckets freely. ## Cost, honestly Bucketing is only economical where keys repeat far more often than the bucket size. Suppose N is 3. A key that is requested a hundred times inside the TTL costs three generations instead of a hundred — an excellent trade. A key requested four times costs three instead of four, which is almost nothing saved for a good deal of added complexity. And on a long tail where most keys are seen once or twice, bucketing costs exactly what no caching would cost while creating the impression that a cache is doing work. Before enabling it, look at the distribution of repeats per key rather than the aggregate hit rate. ## Interaction with seeds and TTL Bucketing composes with the other cache controls rather than replacing them. `Helicone-Cache-Seed` still partitions the key space, so a per-tenant seed multiplied by a bucket of three means three generations per tenant per key — worth doing the arithmetic on before shipping. The `Cache-Control` TTL still governs expiry, and when entries expire the bucket refills from scratch, paying the warm-up cost again on every TTL boundary. A short TTL and a large bucket together are close to no caching at all. ## When to reach for it Use bucketing for user-facing, open-ended generations that repeat heavily and where sameness is noticeable: suggested prompts, example outputs, greeting or summary text on a popular shared object. Avoid it anywhere a machine consumes the output, anywhere a test asserts on it, and anywhere your repeat rate per key is not comfortably above the bucket size.