A reader group's gap is flat, rising steadily, or sawtoothing over a one-hour observation window — what does each shape mean?

## Why the shape beats the value A gap of four hundred thousand records means almost nothing on its own. The same figure is a healthy system that has always sat there and a failing system that passed through it twenty minutes ago on the way up. What separates those two is the **first derivative** — whether the number is going up, down or nowhere over an observation window. The underlying model is a level with two taps. The arrival rate fills it, the drain rate empties it, and the gap is the level. Everything below follows from that. ## Flat and non-zero The level is constant, so the two rates are equal. That is a steady state, not a fault. A standing gap has ordinary causes: - Readers fetch in batches, so a batch's worth is permanently in flight. - There is a fixed pipeline delay between a record arriving and the position moving past it. - The reading side was deliberately provisioned to keep up, not to be ahead — being exactly at the newest record is not a goal anyone should pay for. What matters is that it is flat *at a height you have seen before*. A flat curve at ten times its usual height is a level that rose earlier and then stabilised, which is a different story from a level that never moved. ## Rising in a straight line A constant slope means a constant deficit: the drain rate is lower than the arrival rate by exactly the slope. Whether that is the readers slowing down or the writers speeding up is not visible in this curve alone — you need the arrival rate on the same window to tell them apart, and that distinction matters because the two have different owners. A straight rise also has a property worth saying out loud: it does not self-correct. A level with a constant deficit grows without bound until something changes, and on platforms that remove old records after a fixed period, what eventually changes is that the oldest unread records are deleted before anyone reads them. ## Rising with a curve Two curved rises show up and mean opposite things: 1. **Accelerating** (getting steeper) — the deficit itself is growing. This is often a feedback effect: the reading side is degrading as it goes, or the arrival rate is still climbing toward a peak. 2. **Decelerating** (flattening out) — the deficit is shrinking; the system is heading toward a new, higher steady state rather than toward infinity. An accelerating curve is the one to take seriously fastest, because extrapolating a straight line understates where it is going. ## Sawtooth A regular climb followed by a sharp drop, repeating on a period, means one side is bursty. Both causes are common and they are distinguishable: | What you see | Likely cause | How to tell | |---|---|---| | Climb is fast, drop is near-vertical, period is regular | The reading side runs on a schedule and clears the accumulation each time it wakes | The period matches a schedule, and the drop is faster than any plausible arrival pause | | Climb is near-vertical, then a steady decline | Records arrive in batches; the reading side drains continuously between arrivals | The step up matches a known upstream batch job's size | A sawtooth is not by itself a fault. The two readings to take from it are the **peak** (does the height at the top of each tooth stay constant, or creep up tooth by tooth?) and the **age at the peak** (how stale is the oldest record at the worst moment of each cycle?). A sawtooth whose peaks creep upward is a rising curve in disguise — the drain is not quite clearing each batch. ## Falling, and flattening at a ceiling A falling curve is recovery, and its slope says how fast. The useful arithmetic is comparing that slope against the arrival rate, because a curve that falls only while arrivals are low is not really recovering — it is waiting for the night. The shape that misleads most often is a rise that stops abruptly and becomes flat at a large value. It reads like the problem stabilising. On platforms that delete records after a retention period, it can instead be the ceiling: records are ageing out at the oldest end exactly as fast as they arrive at the newest, so the count cannot grow any further. The tell is the age reading alongside it — an age pinned at a constant value close to the retention span, rather than still climbing, means records are being lost rather than accumulating. ## Per-unit against aggregate One caution that applies to every shape above: where a stream is divided into parts and each part carries its own gap, the curve you are looking at is usually a sum. A sum can be flat while its terms are not, so a shape read off the aggregate is a first reading and not a final one.