skip to content

How does inserting keys in ever-increasing order — a sequence-generated id or an insert timestamp — differ inside a B+tree index from inserting randomly distributed keys such as random UUIDs? Consider split behaviour, page density, write I/O and concurrency.

level: seniorimportance: must knowfreq 55%

answer

  1. Increasing keys → rightmost leaf only, lopsided split, dense
  2. Random keys → splits everywhere, ~2/3 density
  3. Sequential = tiny write working set, stays cached
  4. Random = read-before-write once index exceeds memory
  5. Sequential's cost is the hot-page latch; time-sortable ids are the middle ground

basics

~20 s

Increasing keys always land at the rightmost leaf, so engines split it lopsidedly (nearly all entries stay), pages fill densely, and writes stay concentrated and sequential. Random keys split pages all over the index, settling near two-thirds density with scattered, cache-hostile writes — but they avoid the rightmost-page contention hot spot.

solid answer

~1 min

**Ever-increasing keys.** Every new key exceeds all existing keys, so it always targets the rightmost leaf. A midpoint split there would leave the left half permanently half-empty, since nothing will ever be inserted into it again — so implementations detect the pattern and split lopsidedly, leaving the old page nearly full and starting a fresh page for the new entry. The result is high density, a tiny write working set (the rightmost leaf and its ancestors stay cached), and mostly sequential writes. The downside is a **hot spot**: every concurrent inserter contends on the same page and its latch, and in a clustered organization on the same physical region. **Random keys.** Inserts scatter across the whole key space. Pages split at the midpoint everywhere, so steady-state density settles near two-thirds. The write working set is the entire index, so the buffer cache holds dirty pages across the whole structure, checkpoints write far more pages, log and replication volume rise, and once the index exceeds memory every insert costs a random read first. Sortable identifiers (time-ordered UUIDs) recover most of the locality while keeping global uniqueness; hash-partitioning or a leading shard column spreads a monotonic hot spot across several insert points.

code

text · 9 lines
text
sequential key (id, created_at)
  leaf chain: [....][....][....][....][###]  <- every insert here
  split shape: ~90/10, old page stays dense
  dirty pages per 1000 inserts: a handful

random key (uuid v4)
  leaf chain: [#.#.][.##.][#..#][.#.#][##..]  <- inserts scattered
  split shape: ~50/50 everywhere, steady-state ~2/3 full
  dirty pages per 1000 inserts: up to 1000 distinct pages

go deeper

for a junior

Know that increasing keys always insert at the end of the index while random keys insert everywhere, and that the first keeps pages fuller.

for a middle

Explain the lopsided rightmost split, the ~2/3 steady-state density under random inserts, and the difference in how many pages get dirtied.

for a senior

Reason about the write working set and the memory crossover, checkpoint and log amplification, physical ordering, and the contention trade-off in the other direction.

for a principal

Own the key-design decision: recommend time-sortable identifiers as the default resolution, and know when to deliberately spread inserts for concurrency and what that costs downstream.

## The one fact that drives everything An index entry must go into the single leaf whose key range covers its key. So the *distribution of new key values* decides where in the structure the write traffic lands. Everything else — split shape, density, cache behaviour, I/O pattern, contention — follows from that. ## Ever-increasing keys A sequence-generated integer or an insertion timestamp produces keys greater than every key already present. Consequences: **Only the rightmost leaf is ever written.** The rest of the index is read-only from the insert path's perspective. Its interior pages are never dirtied; only the rightmost leaf and the ancestors on its path change. **Splits are deliberately lopsided.** A midpoint split at the right edge would be wasteful: the left page would sit half-empty forever because no future key can fall into its range. Implementations therefore detect an append pattern and split so that the existing page keeps nearly all its entries and the new page starts almost empty, ready to absorb the continuing stream. This is often described as a 90/10 or a 100/0 split. **Density stays high.** Because pages are filled once and never revisited, the index approaches the density it would have if freshly built. **The write working set is tiny.** The rightmost leaf plus a handful of ancestor pages, which stay resident in the buffer cache regardless of index size. Inserts do not need to read a page from storage first, because the page they need is already hot. This property holds even when the index is far larger than memory — a decisive advantage. **I/O is sequential-ish.** New pages are allocated in order, so the index grows contiguously and the leaf chain stays close to physical order, which keeps later range scans efficient. **The cost is contention.** Every concurrent inserter needs the same page. They queue on its latch; the split of that page briefly serialises them harder. In a clustered organization where the table itself is ordered by the key, the contention extends to the physical block being appended to and to the buffer holding it. On a many-core machine with a high insert rate, this hot spot — not I/O — becomes the throughput ceiling. ## Randomly distributed keys A random UUID or a hash-derived key spreads inserts uniformly across the key space. **Splits happen everywhere, at the midpoint.** There is no append pattern to detect, so pages split evenly. Under sustained random insertion the long-run average density settles around two-thirds of capacity — meaning the index needs roughly 50% more pages than a densely packed one for the same entries, with the corresponding penalty on every range scan and on cache effectiveness. **The write working set is the whole index.** Any leaf can be the target of the next insert. While the index fits in memory this is survivable; once it exceeds memory, each insert may first have to *read* the target leaf from storage — a random read on the critical path of a write. Insert throughput can fall by an order of magnitude at that crossover, and the fall is abrupt rather than gradual. **Checkpoint and log pressure rise.** Dirty pages are scattered across the whole index, so each checkpoint writes far more distinct pages. Engines that log a full page image the first time a page is modified after a checkpoint pay that cost across many more pages, inflating log volume, replication traffic and backup size. **Physical ordering erodes quickly.** New pages come from wherever free space is, so the leaf chain's logical order diverges from physical layout, degrading large ordered scans. **But contention is spread.** Concurrent inserters touch different pages, so there is no single hot latch. For very high concurrency this is a genuine advantage of random keys, and it is the honest counterweight to everything above. ## Choosing, and the middle ground The usual advice — prefer sequential keys — is right for most systems, because density, cache residency and sequential I/O dominate. But it is a trade, not a law: - **Time-sortable identifiers.** Identifier schemes that put a timestamp in the high-order bits and randomness in the low-order bits (time-ordered UUID variants, ULID-style ids) give near-sequential insert locality while keeping the global uniqueness and client-side generation that made random UUIDs attractive. This is the standard resolution and should be the default recommendation when random UUIDs are on the table. - **Deliberate spreading.** When the rightmost-page hot spot is the real bottleneck, spread it on purpose: a leading partition or shard column that splits inserts across N insert points, hash partitioning, or (in engines that offer it) transforming the key so consecutive values land apart. Each trades some scan locality for concurrency. - **Narrow keys matter too.** A 16-byte random key does not merely scatter writes; it also halves the fanout relative to an 8-byte key, adding entries per page pressure and potentially a tree level, and it inflates every secondary index that carries the primary key as its row reference. ## Diagnosing which one is hurting you Sequential-key trouble looks like contention: high concurrency, waits concentrated on one page or buffer, throughput flat as you add cores. Random-key trouble looks like I/O: insert latency stable while the index fits in memory, then a cliff; a read-per-write pattern on the write path; checkpoint write volume far exceeding the logical data written; and an index noticeably larger than its live contents warrant.

  • A team wants client-generated globally unique identifiers but is worried about random-key insert behaviour. What do you recommend?
    A time-sortable identifier: a UUID variant or ULID-style scheme that places a timestamp in the high-order bits and randomness in the low-order bits. Ordering by the high bits restores insert locality — inserts cluster at the right edge, pages stay dense, and the write working set stays small — while the random low bits preserve uniqueness and allow generation without a round trip to the database. It is strictly better than random UUIDs for this workload, at the cost of leaking approximate creation time.
  • When would you deliberately choose a key that scatters inserts?
    When the rightmost-page hot spot is the measured bottleneck rather than I/O: very high concurrent insert rates where sessions queue on a single leaf latch and throughput stops scaling with cores. Spreading inserts across many pages removes that serialization point. It is a deliberate trade of density, cache residency and scan locality for concurrency, and it is worth making only when you have evidence that contention, not I/O, is the limit.
  • Why does insert throughput on a random-key index fall off a cliff rather than degrading smoothly?
    While the index fits in the buffer cache, every insert's target leaf is already in memory, so the write costs no read. Once the index exceeds available memory, a random insert increasingly finds its target page evicted and must read it from storage before it can be modified — a random read on the critical path of every write. The transition happens over a narrow range of index sizes, so the throughput curve looks like a cliff rather than a slope.

Filing new documents by date into the last drawer of a cabinet versus filing them by a random reference number: the first drawer stays open and full, the second means walking to a different drawer every time and leaving gaps everywhere.

saying these in an interview costs you the question

  • "Sequential keys are always better" — they concentrate contention on one page, which can cap throughput on high-concurrency inserts
  • "Random UUIDs are fine because the index is a tree and trees handle any order" — correctness is fine; density, cache residency and write amplification are not
  • "A UUID primary key only costs extra bytes" — it also scatters writes, lowers fanout, and inflates every secondary index that carries the primary key
  • "Rightmost splits are 50/50 like any other split" — engines detect the append pattern and split lopsidedly on purpose
  • "Making the key sequential fixes an insert bottleneck" — if the bottleneck is latch contention, a sequential key makes it worse

context