A peak-concurrency sweep over a production session log never returns to zero — what went wrong?

## The symptom is the diagnostic A delta sweep has a built-in checksum that most people never use. Every well-formed interval contributes exactly `+1` and exactly `-1`. Therefore, over a complete log: - the running sum **must** be zero after the last event, and - the running sum **must never** go negative at any point. When either property fails, the scan is not wrong — the input violated the contract the scan assumes. Recognising this instantly, rather than re-reading the accumulate loop, is what separates a senior answer here. ## The two failure directions and what each means **Ends above zero — unmatched starts.** Every one of these is a `+1` with no partner: - *Still-open sessions.* Users logged in when the log was cut. Perfectly legitimate data, and the most common cause. - *Crashed or disconnected clients.* The logout event was never emitted at all. A session that should have lasted eight minutes now inflates concurrency for the rest of the window. - *Duplicate start events.* At-least-once delivery in the ingestion pipeline replays a login. The peak is inflated by the duplicate count, and the residual tells you how many. - *Filtering asymmetry.* A downstream filter (a bot exclusion, a region predicate, a schema-version check) that happens to match starts and ends at different rates. This one is nasty because the log looks intact. **Dips below zero — unmatched ends.** These are `-1` events whose `+1` lies outside the slice: sessions that began before the window you loaded. If you are sweeping a single day out of a long-running service, this is guaranteed, not exceptional. It also means your peak is *under*-counted, because the sessions carried in from before midnight are invisible to the scan. ## The repairs, and the policy question underneath them There is no universally correct fix, only an explicit choice you must be able to defend: 1. **Deduplicate first.** Group by session identity, keep one start and one end. This is the only repair that is unambiguously right, because a duplicate is not data. 2. **Reject impossible records.** End coordinate before start coordinate is either clock skew across machines or a timezone mix. Such a record makes the running sum go negative locally and can push the peak either way; quarantine and count them. 3. **Decide what an open session means.** Clamping an unterminated session to the window end says "it was still open when we stopped looking", which is right for still-live sessions and wrong for crashed ones — a crashed client's session was over long before the window ended. Dropping it says the opposite. Many pipelines split the difference with a maximum-duration cap: any session still open after `T` is closed at `start + T`, on the grounds that a real session that long is a leak. Whatever you choose, it is a *product* decision about what "concurrent" means, and it belongs in writing next to the metric. 4. **Seed the carried-in count.** For a windowed sweep, initialise the running sum with the number of sessions known to be open at the window start, instead of zero. Without that seed the first hours of every window under-report, and the daily peak metric quietly depends on where you drew the window. ## Making it permanent All of this is one extra linear pass at most, and most of it fits inside the existing scan: track `minimum` alongside `maximum`, count repairs, and emit them with the result. Good practice is to fail the job loudly when the residual exceeds a threshold — a handful of open sessions is normal, ten percent of the log is an ingestion incident — and to publish the repair counts next to the peak so a reader can tell a real traffic record from a data-quality artefact. ## What a weak answer looks like The weak answer starts debugging the accumulation: adding branches, clamping the sum at zero, or switching the tie order in the hope that it helps. Clamping is actively harmful — it hides the negative dip that was telling you sessions were carried in, and it converts a loud data problem into a quietly wrong number. The tie order matters only when two boundaries share a coordinate; it cannot make a running sum drift monotonically upward. The strong answer names the invariant first, uses the residual as a measurement of how broken the input is, and only then talks about policy — because the number that ends up on a dashboard is only as trustworthy as the event contract behind it.