skip to content

Go's write barrier slows an in-memory index service that rewrites a pointer-heavy tree, but only while marking — how do you confirm that and reduce it?

level: seniorimportance: nice to knowfreq 20%

answer

  1. the dip has a period, and it is not traffic
  2. read the generated code, do not guess
  3. count the stores one update performs
  4. a link does not have to be a pointer
  5. no pointer words means nothing to trace

basics

~20 s

Confirm by showing the dip lines up with collection cycles, then disassemble the update path and count the barriered pointer stores per update. Reduce it by rewriting fewer links per update and by replacing child pointers with indices into one node slice, which removes both the barrier and the marker's scan work.

solid answer

~60 s

First establish the pattern: throughput sags while a cycle is marking and recovers between cycles, which points at work that only exists during marking. Then look at the update path itself — `go tool objdump` on the binary, or `go build -gcflags=-S`, shows each pointer store into a heap node preceded by a test of the runtime's barrier flag and a branch, so you can count how many barriered stores one logical update performs. Benchmark that path against a pointer-free variant of the same structure to size the gap. The fixes are representational, not tuning: perform fewer link rewrites per update, for example by building a replacement subtree and publishing it with a single pointer store instead of rewriting a whole parent chain; and cut pointers out of the node itself by holding nodes in one `[]node` and using int32 indices for children. That removes those barriers entirely and, because the slab has no pointer words, gives the marker nothing to trace there either. The cost is readability and manual lifetime management, so measure before you commit.

code

go · 10 lines
go
// Pointer-linked: each child rewrite is a barriered store.
type node struct {
	children [4]*node
}

// Slab-and-index: children are offsets into one []packed.
// No pointer words, so no barrier and nothing to trace.
type packed struct {
	children [4]int32
}

go deeper

for a junior

You would not be asked to lead this, but understand the basic claim: pointer-heavy structures make the collector's job bigger, and structures built from plain numbers give it nothing to trace.

for a middle

Be able to explain why the slowdown appears only while marking, and why swapping pointer links for indices removes both the barrier on the write and the marker's work on the read side.

for a senior

Show the investigation, not just the fix: establish phase correlation, get evidence from the generated code, size it with a benchmark against a pointer-free variant, and change the representation behind an unchanged interface.

for a principal

Own the tradeoff you are asking the team to live with — index-based links buy throughput and cost type safety, debuggability and manual lifetime management, so decide whether the measured win justifies that on a structure other people will maintain.

## The symptom An in-memory index service holds a tree that is rewritten in place on every update. Under load, latency is fine most of the time but the write path visibly slows in bursts, and the bursts line up with garbage-collection cycles rather than with traffic. Between cycles, the same code is fast again. That shape — a regression that exists only while marking is in flight — points at work that is switched on for the mark phase, and on a pointer-write-dense structure the leading suspect is the write barrier. ## Confirming it rather than assuming it **1. Establish the phase correlation.** The dip must track collection cycles, not request mix, heap growth, or an unrelated periodic job. If throughput drops and recovers in step with cycles, you have a mark-phase cost. If the drop persists between cycles, look elsewhere. **2. Look at the generated code for the update path.** `go tool objdump` on the built binary, or compiling with `go build -gcflags=-S`, shows each pointer store into a heap object as a test of the runtime's write-barrier flag, a branch, and a slow path — sitting right next to single-instruction stores for pointer-free fields in the same function. This is the concrete evidence: you can count the barriered store sites and multiply by how many of them one logical update executes. An update that relinks a path from a leaf to the root performs one barriered store per level; at depth 20 that is 20 per update, not one. **3. Bracket the cost with a benchmark.** Benchmark the update path, and benchmark a variant of the same structure whose links are not pointers. Run with `-benchmem` so allocation differences do not get confused with barrier cost. The delta between the two variants, measured under a workload that keeps a cycle running, is the number you are actually trying to move. ## The levers, in order of how much they change **Do fewer pointer stores per update.** Barrier work is per barriered store, so halving the number of link rewrites halves it. Concretely: update a node in place rather than copying its whole parent chain; batch a run of updates and relink once; build a replacement subtree off to the side and publish it with a single pointer store into the parent. **Take pointers out of the representation.** This is the big one. Hold all nodes in one `[]node` and make child links `int32` indices into that slice. Every child rewrite becomes a plain integer store with no barrier at all — and, because a slab of pointer-free structs contains no pointer words, the collector has nothing to trace inside it either, so the mark phase gets cheaper as well as the writes. You get the second win for free, and on a large index it is often bigger than the first. **Keep pointer-bearing fields out of the hot slab.** If nodes must carry strings or interface values, consider holding those in a parallel structure so the frequently rewritten part stays pointer-free. ## The levers that are not levers - You cannot switch the write barrier off. It is not exposed and it is not optional; a program that skipped it would free live objects. - Bypassing the barrier with `unsafe` pointer writes is not an optimisation, it is heap corruption waiting for a load spike. - Adding synchronisation, pooling, or forcing a collection with `runtime.GC()` does not touch barrier cost at all, and forcing collections makes the marking window larger, not smaller. ## The tradeoffs to state out loud Index-based structures give up compile-time type safety on links, make debugging harder (an `int32` in a dump tells you less than a pointer), and hand you manual lifetime management: a node removed from the tree is not freed until you reuse or shrink the slab, so a workload with heavy churn can hold more memory than the pointer version did. They also serialise nicely and have better locality, which is often a second, unrelated win. Because of that, the honest sequence is: confirm the phase correlation, measure the barriered-store count in the hot path, prototype the pointer-free variant behind the same interface, and only adopt it if the benchmark says the win is worth the readability. `The one making it faster without changing behaviour` should be able to show a before/after benchmark and an unchanged test suite, not just an argument about how the runtime works. ## What a strong answer sounds like Name the phase-correlated symptom, produce evidence from the generated code rather than intuition, size it with a benchmark against a pointer-free variant, and then propose changes to the data representation — fewer pointer stores, or no pointers at all on the hot links — while being explicit that the collector's behaviour itself is not something you get to turn off.

  • Why does replacing child pointers with indices help twice over?
    Once on the write side: an int32 store carries no reference, so no barrier code runs at all. And once on the collector's side: a slab of pointer-free structs has no pointer words in it, so the marker never has to trace through it, which shortens the mark phase itself. Both wins come from the same representational change.
  • How would you rule out that the regression is something other than barrier work?
    Check that the slowdown is phase-correlated — present while marking, gone between cycles. Then confirm from a disassembly that the hot path really does perform many barriered pointer stores, and benchmark the identical algorithm over a pointer-free representation. If the pointer-free variant shows no phase-correlated dip, the barrier was the cost.
  • What do you lose by moving a hot tree onto a slab with integer links?
    Type safety on links, debuggability, and automatic reclamation. An int32 index can point at a recycled slot, a dump is harder to read, and nodes removed from the tree are not freed until you reuse or shrink the slab — so a high-churn workload can retain more memory than the pointer version did. Measure before adopting it.
  • Is forcing a collection between batches a reasonable way to avoid barrier cost on the hot path?
    No. Calling `runtime.GC()` starts a cycle rather than avoiding one, so it increases the fraction of time the barrier is active. If updates arrive continuously there is no quiet window to hide a cycle in anyway. The durable fix is fewer pointer stores, not trying to time the collector.

saying these in an interview costs you the question

  • Proposes disabling the write barrier
  • Suggests unsafe pointer writes to skip the barrier
  • Blames the barrier without checking phase correlation
  • Calls runtime.GC between batches to dodge marking
  • Reasons only from a mental model, never from generated code or a benchmark
  • Ignores that index-based links change memory lifetime