In a load run held steady for hours, why assert on the growth trend rather than the peak?
answer
- A direction, not a height
- Two samples cannot show a plateau
- Filling to a limit is not leaking
- Trend the reclaimed floor, not peaks
- A slope needs a comparator to mean anything
basics
~20 sA peak says how high a figure got; accumulation is a direction. Measure retained memory, open handles, cached entries and disk use as a slope across the hold, then compare that slope with a period expected to be level.
solid answer
~50 sA single value cannot separate a structure that filled to its limit early and stayed there from one climbing steadily that has not arrived yet. Sample the counters on a fixed cadence for the whole hold and fit a slope on the part after filling has finished; for memory that is reclaimed periodically, trend the floor it returns to rather than the sawtooth peaks. Then give the slope something to be compared against, because a bare percentage per hour means nothing: the spread of the same slope across repeated runs, the same counter divided by completed work so a throughput change is not mistaken for growth, and the remaining headroom, which turns the slope into hours before it hurts. Trend several counters — memory, handles, cached entries, on-disk bytes, queue depth — since they fail differently.
code
pseudocode · 18 linessamples = collect_every(minutes = 1, counters = [
retained_memory_floor, open_handles, cached_entries,
on_disk_bytes, queue_depth, completed_operations
])
flat = samples.after(filling_finished) # bounded structures have stopped rising
for counter in flat.counters_except(completed_operations):
per_hour = slope(flat[counter], against = elapsed_hours)
per_work = slope(flat[counter], against = flat.completed_operations)
if per_hour <= noise_band[counter]:
verdict = "inside the noise of repeated runs"
else if per_work is about 0:
verdict = "advances with the clock, look outside the system"
else:
verdict = "accumulating per unit of work"
hours_to_ceiling = headroom[counter] / per_hourgo deeper
Be ready to explain why one memory reading cannot show a leak: a direction needs several samples across time. Know that some structures are supposed to grow at the start of a long run and then stop.
Explain how the slope is actually fitted — regular sampling, the reclaimed floor rather than the peaks, and the portion of the hold after bounded structures finished filling. Expect to be asked which counters you would trend besides memory.
Show the comparisons that make a slope mean something: its noise band across repeated runs, the same figure per unit of completed work, and the headroom that converts it into hours before it hurts.
Own what every long hold trends by default, so growth series exist without anyone requesting them, and decide how much residual accumulation the organisation accepts when hours-to-breach is long but finite.
## A peak answers a different question A single figure, whether the highest value seen during the hold or the value at the moment it ended, answers "how big did this get?". Accumulation is not a size; it is a direction. Two systems can finish a long hold at the same memory figure and be in opposite states: one filled a cache to its configured limit in the first ten minutes and has been level ever since, the other has been climbing steadily and simply has not arrived yet. A peak cannot tell those apart. A slope can, and it can also say roughly when the second one arrives. That is why the assertion for a run held at a steady rate for many hours is written about a trend rather than a value: retained memory, open handles and connections, entries held in caches and maps, thread or task counts, on-disk bytes in logs, temporary files and spooled data, and queue or backlog depth. ## Reading the slope - **Sample on a fixed cadence for the whole hold.** Two points, one at each end, cannot distinguish a straight climb from a rise that flattened after an hour. - **Trend the reclaimed floor, not the sawtooth.** Where memory is reclaimed periodically, the peaks move with how recently reclamation ran. The low point that memory returns to after each reclamation is what is genuinely being retained. - **Fit on the part of the hold where filling is over.** Pools, caches and buffers are supposed to rise and then stop; a slope fitted across the fill phase reports growth that is by design. - **Trend several counters, because they fail differently.** A leak that never touches memory shows up as connections or file handles that are opened and never returned. - **Express the result two ways**: per hour of hold, and per unit of completed work. ## What a slope has to be compared against A number like "two percent an hour" means nothing on its own. It becomes a finding only against a comparator, and each comparator rules out a different innocent explanation. | Comparator | What it rules out | |---|---| | The spread of the same slope across repeated runs | A rise that is measurement variation rather than build-up | | A control period, or a control run, at a much lower rate | Growth that advances with the clock rather than with work done | | The same counter divided by completed work | A rise that is only more work being done, or a fall that is only less | | The remaining headroom to the structure's ceiling | Converts a slope into hours-before-it-hurts, which is the figure people act on | | The same hold shape before the change under test | Whether this release moved the slope at all | The middle two are the ones most often skipped. If throughput drifted during the hold, a rising total can simply reflect more work; dividing by completed work makes that confound visible. And if the same slope appears in a control period at a tenth of the rate, the growth is not caused by the work at all and the search should move outside the system. ## Bounded and unbounded growth look identical early The distinction the trend exists to draw is between growth that decelerates towards a ceiling and growth that does not. Early in a hold they look the same, which is why the shape of the later part of the window carries the information: - If the slope over the last quarter of the hold is as steep as the slope over the quarter after filling ended, nothing is bounding it. - If each successive quarter is flatter than the last, the structure is approaching a limit and the interesting number is where that limit sits. - A staircase, where the figure plateaus after each reclamation or eviction cycle but at a higher level than the previous plateau, is a slow build-up wearing the costume of a bounded one. Compare plateau to plateau, not point to point. ## Reporting a trend so someone else can use it The report of a long hold should carry, for every counter trended: the sampling cadence, the window over which the slope was fitted, the slope per hour and per unit of completed work, the noise band the slope was compared against, and the extrapolation to the ceiling with the assumption of linearity stated out loud. That is a handful of lines and it makes the result reusable — the next person can compare their hold against yours instead of starting again, and a reader can see immediately whether a rise of two percent an hour was worth anyone's attention or was simply inside the noise of the instrument.
- What do you trend when memory is reclaimed periodically and the chart is a sawtooth?Trend the low points — the level memory returns to after each reclamation — because that floor is what is genuinely retained. The peaks move with how recently reclamation ran and look alarming even when nothing accumulates. Rising floors across the hold mean something is held; level floors mean the sawtooth is normal.
- A counter rises two percent an hour across the hold. What do you need before calling that a defect?Three things. The spread of the same slope across repeated runs, so the rise is bigger than the instrument's noise. The slope per unit of completed work, so a throughput change is not being read as growth. And the remaining headroom, which converts the slope into hours before it hurts — two percent an hour against a ceiling three days away means something quite different from one three months away.
One glance at a bathtub's water line tells you nothing. With the tap at a constant flow, what matters is whether the level is still rising an hour later, and how far it is from the rim.
saying these in an interview costs you the question
- Reads the highest memory figure and calls it healthy
- Samples only at the start and the end of the hold
- Calls a cache filling to its limit a leak
- Treats any upward slope as a defect without a noise band
- Trends the sawtooth peaks instead of the reclaimed floor