A per-IP counter hash table's p99 spikes periodically while mean latency stays flat — how do you confirm resizing is the cause?

## The shape you are looking at A rate limiter keyed by client address inserts a new entry the first time it sees each address. As distinct addresses accumulate, the table crosses its load threshold and doubles, re-placing every stored entry on the unlucky insertion that triggered it. That one operation runs for as long as it takes to walk hundreds of thousands of entries; every other operation that hits the same table behind a lock waits for it. The result on a dashboard is a **sawtooth in the tail metric and nothing at all in the mean**. If one operation in 500,000 takes 40 ms while the rest take 200 ns, the average moves by well under a microsecond — invisible — while p99.9 and max jump by orders of magnitude. "Tail moves, mean doesn't" is itself evidence: it says a tiny number of operations are enormously slow, which is the signature of a rare bulk operation rather than of general slowness. ## Confirming it, not asserting it The goal is a correlation an SRE can check without trusting your intuition. 1. **Export the table's state.** Entry count and capacity as gauges. Capacity is a step function; every step should line up with a spike. 2. **Time the resize itself.** Wrap the growth path so it emits an event carrying duration, entries moved, and old/new capacity. This turns the hypothesis into a measured fact and gives you the spike's magnitude in advance. 3. **Overlay and check three things.** Do spikes coincide with capacity steps? Does spike magnitude scale with entries moved — each spike roughly twice the previous one? Do the gaps between spikes roughly double? 4. **Check which operations are slow.** Only paths that can trigger growth should spike. If read-only lookups spike identically, growth is not your culprit. 5. **Watch for the ending.** Once the population of distinct addresses saturates, growth stops and so do the spikes — without any deploy. That behaviour is very hard to explain with any other cause. ## Ruling out the usual suspects The doubling interval is the discriminator worth leading with, because the competing explanations mostly have *fixed* periods: - **A scheduled task** — a flush, a metrics roll-up, a cron — repeats on a constant interval. Resize spikes do not; they get rarer geometrically. - **Memory reclamation pauses** correlate with allocation volume and are often multi-threaded and process-wide, hitting operations that never touch this table. A resize is single-structure and single-threaded. Note that resizing *feeds* reclamation pressure by discarding a large array each time, so the two can appear together — measure the resize event itself rather than inferring from allocation graphs. - **Lock contention** rises with concurrency, not with table size, and shows up as a broad p99 lift rather than isolated spikes. During a resize the effect is real but it is a *consequence* of the transfer, not an independent cause. - **A downstream dependency** shows correlated latency at that dependency's own metrics; the table's capacity gauge would show no step. ## Explaining the sawtooth to a skeptic The question an SRE will ask is "why is it getting less frequent but worse?" The answer is one sentence: each resize doubles the capacity, so it takes twice as long to fill again, and there are twice as many entries to move when it does. Frequency halves, magnitude doubles, total work per unit of growth stays flat. The chart looks alarming precisely because it is asymptotically well-behaved — that is the amortized bound rendered as a picture. ## Then decide whether to fix it Diagnosis is not automatically a work item. If the address population saturates within minutes of a deploy, the spikes are a startup transient and the correct action may be to exclude the warm-up window from the SLO. If the spikes are recurrent because the key space genuinely churns — addresses arriving and never being reused — the table is growing without bound and the real defect is the missing expiry, not the rehash. If they breach the budget in steady state, the mitigations are pre-sizing from a known address population, splitting the counters across several fixed sub-tables so each transfer moves a fraction of the entries, or bounding the table with eviction. Choosing among those is a design conversation; confirming the cause with the capacity gauge is the part that must come first.