What does sys.intern() do to a Python string, and when is calling it worth it?

### Two different mechanisms wear the same word "Interning" in CPython covers two things that are easy to conflate. The first is **automatic sharing**, which you never ask for: the compiler stores each distinct constant of a compilation unit once, string literals that look like identifiers (ASCII letters, digits and underscores) are placed in the interpreter's intern table at compile time, and the integers from -5 through 256 are preallocated once at start-up. The second is **explicit interning** with `sys.intern()`, which is the only tool you have for strings that did not exist when the code was compiled. That second category is where the memory goes. A string produced at runtime is always a brand new object: the result of `str.split()`, of decoding bytes, of concatenating two variables, of a parser handing you a field name. CPython does not check whether an equal string already exists — that check would cost a hash and a lookup on every string creation, which would be a bad trade. So a loop that reads a million records and builds a dict per record allocates a million separate copies of every key, each with its own object header, its own cached-hash slot and its own character buffer, all holding identical bytes. ### What `sys.intern()` actually does `sys.intern(s)` hashes the value, looks it up in the interpreter-wide intern table, and returns the object already stored there if there is one; otherwise it stores `s` and returns `s`. Rebinding your name to the returned object means the duplicate you just built has no references left and is freed immediately by reference counting. The net effect on a repetitive data set is one object per *distinct* value rather than one per *occurrence*. There is a second, smaller win. String equality in CPython starts with a pointer check, so comparing two interned strings that are equal answers on the first instruction instead of walking the characters. Dictionary lookups benefit on collisions and on the final key comparison. Do not oversell it: every `str` caches its hash after the first use whether it is interned or not, so interning does not make hashing cheaper, and the comparison win only shows up in hot loops that compare or look up the same long strings many times. ### The rules and the limits `sys.intern()` accepts an exact `str` only. Passing `bytes` raises `TypeError: intern() argument must be str, not bytes`, and a `str` subclass raises `TypeError: can't intern` — worth knowing when a parsing layer hands you subclassed strings. There is no public way to un-intern a value, and there is no "intern everything" switch. The decision rule is cardinality, not size. Interning pays when a small number of distinct values appears many times: column names, category codes, status flags, symbol names, repeated dict keys. It actively costs you when values are near-unique — request ids, hashes, free text — because you pay a hash and a table probe per call and gain nothing, and the table now carries an entry per value. Measure the repeat rate before you reach for it. On lifetime, CPython 3.14 is friendlier than the folklore. Small integers and a set of statically allocated short strings became *immortal* in 3.12 under PEP 683 — their reference counts never change. Strings you intern yourself at runtime are not in that group: on 3.14 the intern table does not add a counted reference, so such a string is still freed once your last reference goes away. The old claim that "anything you intern lives until the interpreter exits" should not be repeated as a blanket fact on modern CPython. ### The correctness line Interning is a memory optimisation and nothing more. Which values happen to be shared is a CPython implementation detail that varies by version, by build and by other implementations, so no program should behave differently because two equal values turned out to be one object. If you need a guaranteed single instance — a sentinel, a flyweight, a cache — build it yourself with a module-level object or your own dict, where the sharing is a property of your code rather than of the runtime. If profiling says the repeated *keys* rather than the repeated *values* dominate, interning is the cheap 20% fix and a change of record layout is the structural one; that layout question belongs to object overhead, not here. Start with interning because it is a one-line change at the point where the strings are created, measure, and only then restructure.