skip to content

In a proxy, every request thread copies and drops a handle to one shared read-only routing table. Why does that cost throughput?

level: middleimportance: must knowfreq 62%

answer

  1. reading it still writes something
  2. the write is not the table
  3. one word, every core, exclusively owned
  4. atomic read-modify-write per share and per drop
  5. scales with handle traffic, not object size

basics

~10 s

Each copy and drop of a counted handle is an atomic read-modify-write of one shared count, so read-only sharing is still a write workload: cores must take exclusive turns owning that counter's cache line.

solid answer

~40 s

The table's data is read-only, but the bookkeeping beside it is not. Reference counting maintains a count of the owning handles that refer to an object; taking a handle adds one, dropping it subtracts one. Because handles here are taken by many threads, both updates must be atomic read-modify-writes rather than plain increments, and an atomic update of a word requires the core to hold that word's cache line exclusively. So every request performs two exclusive writes to the *same* address as every other request. Uncontended, that is a modest constant; under load it turns one shared word into a serialisation point, and throughput stops improving — or falls — as threads are added. The cost tracks the number of ownership changes, not the size of the table or the amount of data read.

code

pseudocode · 16 lines
pseudocode
// what a counted handle does on copy and on drop
function copy_handle(h):
    atomic_add(h.object.count, +1)      // exclusive ownership of the count line
    return handle(h.object)

function drop_handle(h):
    old = atomic_add(h.object.count, -1)
    if old == 1:                        // this thread removed the last handle
        destroy(h.object)

// one request, two exclusive writes to the same word
function handle_request(req):
    t = copy_handle(shared_table)
    route = lookup(t.object, req.host)  // pure reads, freely shared
    drop_handle(t)
    return route

go deeper

for a junior

Remember the one fact under all of this: a reference count is data that lives with the object, and copying a handle writes it. Read-only sharing of a counted object still touches memory.

for a middle

Explain the mechanics: an ownership change is an atomic read-modify-write, an atomic update needs exclusive ownership of the word's cache line, and so many threads sharing one object serialise on one address regardless of what the object contains.

for a senior

Show that you can size the tax before you profile: ownership changes per request times request rate against one word, then judge whether that is the ceiling. Then name the fix that removes ownership changes rather than making them faster.

for a principal

Frame it as a cost that scales with handle traffic while the benefit — prompt, deterministic release — scales with what the object holds. That framing is what lets you decide per object rather than adopting one reclamation discipline for everything.

## The count is state, and state gets written A reference-counted object carries a **count of the owning handles that currently refer to it**. That count is not part of the object's data; it is bookkeeping the reclamation scheme keeps for the object, either in a header word beside it or in a side table. The rule that makes the scheme work is unavoidable: **creating an owning handle adds one, destroying an owning handle subtracts one**, and the transition to zero destroys the object. That rule has a consequence people rarely expect. **Sharing a read-only object is a write workload.** In the proxy above, nothing about the routing table changes — no route is added, no field is touched, and every thread only reads. Yet each request performs two writes to the same memory word: one when the handler takes its handle, one when the handler drops it. The payload is immutable; the accounting is not. ## Why the update cannot be a plain increment `count = count + 1` is three steps — load, add, store. If two threads run it on the same word, both can load the same old value and both can store the same new one, so one of the two updates disappears. For an ordinary statistic that is a small inaccuracy. For a reference count it is a correctness bug in both directions: - a **lost increment** leaves the count *below* the true number of handles, so the count reaches zero while a live handle still exists and the object is freed underneath its user; - a **lost decrement** leaves the count *above* the true number, so the object is never reclaimed and the memory leaks. So in any scheme whose handles may cross threads, the update must be a single indivisible read-modify-write. That instruction is what costs: to perform it, the core must hold the line containing the count in an exclusive state, which means taking it away from whichever core held it last. ## Where the cost actually lands It helps to separate three quantities that get confused: | Cost | Scales with | Paid by | |---|---|---| | Count update | number of ownership changes | every thread, on the request path | | Destruction | the object's fields and what it owns | the thread that drops the last handle | | Reading the data | bytes touched, cache residency | every reader, but shared lines cost nothing to read | The third row is the one that surprises people. A line that many cores only **read** can sit in all of their caches at once, so wide read sharing of immutable data is close to free. A line that many cores **write** cannot: exclusive ownership moves from core to core, and the transfer is what the threads are waiting on. Reference counting quietly converts the first situation into the second, because it puts a written word next to data that nobody writes. ## Estimating the tax You can put a number on it without any profiler, and interviewers like it when you do: 1. Count the ownership changes per unit of work. In the proxy that is two per request per handler that takes its own handle — and often many more, because passing the handle into helpers, storing it in a request context, and capturing it in a callback each add a pair. 2. Multiply by the request rate to get updates per second against a single address. 3. Compare that with what one cache line can sustain when ownership has to migrate between cores. The ceiling is a property of the hardware, not of your code, and it does not rise when you add threads. When step 3 lands near your target request rate, the count is your scaling limit, and no amount of making the *table* faster will help. ## What this is not - It is **not** a fault of the table's size. A one-field object shared per request costs the same counting tax as a large one. - It is **not** the cost of freeing. Freeing happens once, at the end; the tax here is paid on every share. - It is **not** removed by making the object immutable, which is the whole point of the example. - It is **not** the same as a lock. There is no mutual exclusion in the program, no critical section, and no risk of deadlock; the serialisation happens below the program, in the hardware that owns the line. ## Why anyone pays it Counting buys **prompt, deterministic reclamation**: the object dies at the instant its last handle dies, on the thread that dropped it, with no scan of the heap and no reclamation-wide pause. For a buffer that holds an operating-system resource, or for a footprint that must fall the moment work ends, that promptness is worth real throughput. The engineering question is never 'is counting slow' but 'is this particular object shared often enough that the accounting costs more than the promptness is worth' — and for a process-lifetime read-only table touched by every request, it usually is.

  • If the table is never modified, why does the hardware treat the count differently from the table's data?
    A line that is only read can be resident in every core's cache simultaneously, so read sharing scales. An atomic update must be indivisible, which requires the writing core to hold that line exclusively, invalidating every other copy. The count sits in a written line; the routes sit in read-only lines. Same object, opposite scaling behaviour.
  • Does taking the handle only once per request, instead of in every helper, actually change anything?
    Yes, and it is the cheapest fix available. The tax is per ownership change, so hoisting the handle out of the inner call chain and passing a non-owning borrow downward removes every pair of updates below it. The object is still kept alive for the whole request by the one handle that remains.
  • Is this cost present in a single-threaded program too?
    The instruction cost mostly disappears: with no possibility of a concurrent update the scheme can use plain increments, and an uncontended update of a line the core already owns is close to a normal store. What remains is the instruction count and the extra memory traffic of touching the header on every share, which still shows up in tight loops.

A reference library where everyone may read the same book at once, but each reader must sign the one card at the desk on the way in and on the way out. The reading is parallel; the signing is a queue.

saying these in an interview costs you the question

  • Thinks sharing a read-only object costs nothing because no data is written
  • Assumes an atomic increment costs about the same as a plain one
  • Blames the table's size or its lookup cost rather than counter traffic
  • Says a lock is being taken, and looks for a critical section
  • Believes adding threads must raise throughput because the data is immutable
  • Confuses the cost of destroying the object with the cost of counting