skip to content

What does unique.Make give you when a long-lived Go map holds millions of repeated strings?

level: seniorimportance: nice to knowfreq 20%

answer

  1. keep one copy, point everything at it
  2. the handle is what you store
  3. comparing handles is comparing addresses
  4. the table lets go when nobody holds on
  5. pay the lookup once, at the edge

basics

~20 s

unique.Make interns a comparable value and returns a Handle. All equal values collapse onto one canonical copy, so repeated strings stop duplicating memory, and comparing two handles is a cheap pointer comparison rather than a byte-by-byte one.

solid answer

~50 s

`unique.Make[T comparable](v T) Handle[T]` looks the value up in a process-wide table and returns a `Handle[T]` referring to the one canonical copy; `h.Value()` gives the value back. Store the handle instead of the string and a million rows carrying the same table name hold one string plus a million small handles, rather than a million string headers each pointing at its own bytes. Handles are comparable and usable as map keys, and comparing them is a pointer comparison instead of a length-then-bytes comparison. The table holds its entries weakly, so a canonical value becomes collectable once no handle refers to it any more — unlike a hand-rolled canonicalisation map, which retains everything until you evict it. The cost is a hash and a lookup on every `Make`, so intern at the ingest boundary where values arrive, not in an inner loop, and only when the same values genuinely repeat and live a long time.

code

go · 12 lines
go
type column struct {
	table unique.Handle[string]
	name unique.Handle[string]
}

func newColumn(table, name string) column {
	return column{table: unique.Make(table), name: unique.Make(name)}
}

func (c column) Table() string { return c.table.Value() }

func sameTable(a, b column) bool { return a.table == b.table }

go deeper

for a junior

Recall the idea rather than the API: interning keeps one copy of each distinct value and hands out references to it, saving memory when the same values repeat many times.

for a middle

Explain the shape: Make returns a comparable Handle, Value returns the canonical value back, the constraint is comparable, and handle equality replaces byte comparison.

for a senior

Show when it pays — heavy repetition in a long-lived structure, interned once at the ingest boundary — and cite the weak table as what separates it from a canonicalisation map that leaks. Confirm with a heap profile.

for a principal

Decide whether handles belong in internal representations only or leak into package APIs other teams depend on, and weigh the readability cost of Value() calls against a memory saving you can attribute.

## The problem interning solves A Go `string` is a pointer and a length. Two strings with identical bytes are usually two separate allocations, because each came from its own parse, decode or read. In a long-lived index — a map of schema metadata, a million parsed rows, a graph of nodes each tagged with a category — that duplication dominates: the distinct values number in the hundreds while the copies number in the millions. Interning means keeping exactly one copy of each distinct value and having everything else refer to that copy. ## What the unique package provides The `unique` package offers two things: - `unique.Make[T comparable](value T) Handle[T]` — returns a handle to the canonical copy of `value`, creating it if this is the first time that value has been seen. - `Handle[T].Value() T` — returns the canonical value the handle refers to. `Handle[T]` is itself a small comparable value. Two handles produced from equal values are equal, and two handles produced from different values are not. That is the second benefit after memory: comparing two handles is comparing what amounts to a pointer, rather than comparing lengths and then bytes. In a hot equality check over long strings that can matter more than the memory saved. Handles are also usable directly as map keys. The constraint is `comparable`, so this is not limited to strings: a small struct of a name plus a namespace can be interned too, and often should be, since it collapses two duplicated strings at once. ## Reclamation is the part that distinguishes it The naive alternative is a `map[string]string` you canonicalise through yourself. It works, and it leaks: every distinct value ever seen stays in that map for the life of the process, because the map itself is a strong reference. A metadata cache fed by user-supplied names becomes an unbounded memory growth path. The `unique` package's internal table holds its entries weakly. Once no `Handle` refers to a canonical value any more, that entry becomes eligible for collection and the memory comes back. You get canonicalisation without owning an eviction policy. As with all garbage collection, "eligible" is not "immediate" — the memory returns on the collector's schedule, not at the moment the last handle goes out of scope. ## When it pays and when it does not Every `unique.Make` call hashes the value and looks it up in a concurrent table. That is far more expensive than doing nothing, and it buys nothing when the values are mostly distinct — a million unique request ids interned is a million table entries plus a million lookups plus zero saved bytes. The profile that pays: - **High repetition.** A small set of distinct values appearing very many times. - **Long life.** The handles live in a structure that persists — an index, a cache, a loaded dataset. Interning something you discard microseconds later is pure cost. - **A boundary to intern at.** Call `Make` once where values enter the system — the decode, the scan of a result set, the parse of a file — and pass handles inward. Calling it repeatedly on the same value in an inner loop re-pays the lookup every time. If you are unsure, a heap profile before and after tells you whether the duplication was real, and a benchmark tells you what the lookups cost. ## Ergonomics to expect Code that stores handles has to call `Value()` wherever it needs the string, which is an extra indirection and slightly noisier at the point of use. That is the honest trade: smaller and faster to compare, marginally more awkward to read. It suits internal representations — a parsed schema, an index's node labels — better than types on a public API surface, where handing callers a plain `string` is kinder unless they benefit from the comparison speed too. ## The mental model Think of it as a canonicalising set that cleans up after itself: `Make` is "give me the one true copy of this", `Value` is "give me back what it was", and the entry disappears once nobody is holding it. It is a memory-shape tool, applied at the edges of a long-lived data structure, and it is worth reaching for only after a profile has shown that the duplicated values are actually where the bytes went.

  • How is a unique.Handle better than a hand-rolled map of canonical values?
    Two ways. The handle is a small comparable value you store instead of the string, so equality is effectively a pointer comparison. And the package's table holds entries weakly, so a canonical value is reclaimed once no handle refers to it — a plain `map[string]string` keeps every value ever seen for the life of the process unless you write and tune an eviction policy yourself.
  • When is interning with unique.Make the wrong call?
    When the values are mostly distinct, when they are short-lived, or when `Make` would sit in an inner loop. Each call hashes and looks up in a shared table, so you pay real cost for every value and save memory only where duplication is heavy and long-lived. Confirm with a heap profile before reshaping types around handles.
  • What can unique.Make be applied to besides strings?
    Any type satisfying `comparable`, since that is the constraint on its type parameter. A small struct — say a name plus a namespace — interns as one unit and collapses several duplicated fields at once, and the resulting handle is still a single comparable value you can use as a map key.

saying these in an interview costs you the question

  • Thinks unique.Make compresses or shortens the value
  • Interns every value on the hot path without profiling
  • Assumes interned entries are never reclaimed
  • Compares handles with reflect.DeepEqual instead of ==
  • Expects the memory back the instant the last handle drops