In a log pipeline, when does interning repeated strings pay off — and when does the table become the leak?

## What interning is **Interning** (canonicalization, deduplication) is: maintain a table of distinct values; on constructing a value, look it up; if an equal value is present, discard the new object and use the stored one; otherwise store the new one. Afterwards, every holder of that value references **one** shared object. Immutability is the precondition that makes this legal at all — sharing one object among unrelated holders is only safe if none of them can modify it. The payoff for a log-ingest pipeline is easy to size. Suppose events carry a service name drawn from about 500 distinct values, seen a billion times an hour, at ~30 characters each. Without interning, up to a billion separate 30-character buffers exist transiently; with interning, 500 buffers exist and the billion events hold references. Note carefully what the saving is proportional to: **distinct-count times value size** for what remains, versus **live occurrences times value size** for what is avoided. If your occurrences die immediately after being written to a socket, the memory you "saved" was never live at once and the win is much smaller than the arithmetic above suggests. The real wins come when occurrences are retained: an in-memory index, a batching buffer, a cardinality-tracking aggregator holding millions of event records at once. A second payoff follows: once a field is canonicalized, grouping and comparison on that field can use identity — one reference comparison instead of a character walk — provided every value on both sides went through the table. In a group-by over billions of rows that is a genuine hot-loop win. ## The two costs, and the direction of each **Cost 1 — CPU on every occurrence, savings only on duplicates.** The lookup itself is a hash of the full value plus a confirming content comparison, i.e. O(n) in the value's length, paid *per occurrence*. You pay that billion times to deduplicate down to 500. For a low-cardinality field the trade is excellent. For a field where nearly every value is distinct, you paid the full cost and deduplicated nothing — the table just grew. **Cost 2 — the table retains everything, so garbage becomes a leak.** This is the failure the question is aimed at, and the wrong answer it targets is "interning is free memory savings". A canonical table must hold a reference to each canonical instance, or it could not return it next time. Unless entries are held weakly or evicted, **every distinct value ever interned is retained for the process's lifetime**. Intern a service name and you retain 500 short strings, forever, which is fine. Intern a request identifier, a full URL with query parameters, a user-supplied search term, or an error message with an embedded timestamp, and you have converted a stream of short-lived garbage into monotonically growing permanent memory. The symptom is a process whose memory grows roughly linearly with traffic and never recovers, with the heap dominated by one enormous table — and it typically appears *after* someone "optimised memory" by interning more fields. There is a nastier variant: cardinality that is low in staging and unbounded in production. A field like `tenant_id` looks like a closed set of a dozen values until the platform onboards fifty thousand tenants, and a field like `endpoint` looks closed until someone starts templating identifiers into the path. ## The policy that actually holds up 1. **Measure distinct-count first, on production data**, not occurrence count. Interning is worth it when distinct-count is small relative to live occurrences; a cardinality estimate over a real sample answers this in minutes. 2. **Intern only closed, long-lived sets** — service names, log levels, region codes, metric names, enumerated status values. These are chosen from a fixed vocabulary and their permanent retention is a fixed, small cost. 3. **Never intern anything that carries an identifier, a timestamp, a user's input, or a full path with parameters.** Those are unbounded by construction. 4. **Bound the table.** A fixed-capacity cache with eviction, or weakly-held entries, converts a cardinality surprise from an out-of-memory crash into a graceful loss of deduplication. The cost is that identity comparison is then no longer sound, because an evicted value can reappear as a new object — you must pick one or the other, and that choice is the interesting part of the answer. 5. **Intern at the boundary, once**, where values are decoded, rather than sprinkling lookups through the code. One place to measure, one place to disable. Ecosystems differ on the defaults here in a way worth knowing: Java and C# expose an explicit, permanently-retained canonical pool for strings while also automatically canonicalizing compile-time literals, Python canonicalizes some short identifier-like strings by an unspecified internal rule, and Go leaves deduplication entirely to library code. In every one of them the analysis above is the same; only who owns the table changes. ## Answering it out loud Name the payoff (distinct-count times size, plus an identity fast path), name the two costs (per-occurrence CPU; permanent retention), then give the decision rule: measure cardinality, intern only closed vocabularies, bound the table, and be explicit that a bounded table forfeits the identity comparison.