A service measures how long an operation took by calling a wall-clock/time-of-day API (e.g. the equivalent of `System.currentTimeMillis()`) before and after the operation, then subtracting. Under what circumstances can this produce a negative or wildly wrong duration, and what's the correct fix?

## Two conceptually different kinds of clock Operating systems typically expose two conceptually different kinds of clock, and conflating them is one of the most common sources of subtle timing bugs in production software. | Clock | What it reports | What it is for | |---|---|---| | **Time-of-day** (or "wall") | Reports the current calendar/civil time, such as milliseconds since the Unix epoch. | Its entire purpose is to answer "what is the date and time right now," correlated with the outside world. | | **Monotonic** | Reports elapsed time since some arbitrary, unspecified reference point (often boot time, but implementation-defined). | Its entire purpose is to answer "how much time has passed," with the single hard guarantee that successive reads never decrease. | ## Why wall clocks are corrected from outside The reason this distinction matters mechanically is that wall clocks are, by design, subject to external correction. - **NTP synchronization** can slew the clock gently or step it abruptly if the local clock has drifted too far from the reference. - **A system administrator** can manually change the date. - **A leap second** can be inserted or smeared into UTC. - **Virtualized or containerized environments**: a VM that is live-migrated, paused, or resumed from suspend can have its wall clock reset or catch up in a jump when it resumes. None of these adjustments are bugs - they're the wall clock correctly doing its job of tracking real calendar time - but they mean a wall-clock read taken before an operation and another taken after can, in principle, have the second value smaller than the first if a backward correction happened in between. Subtracting start from end in that scenario yields a negative duration, or in a smaller-magnitude case, silently understates or overstates the true elapsed time without an obviously wrong sign to catch it. ## What a monotonic clock guarantees, and what it gives up Monotonic clocks solve this by explicitly not tracking calendar time at all. On POSIX systems this is `CLOCK_MONOTONIC` (or `CLOCK_MONOTONIC_RAW` for a version unaffected even by NTP's rate-slewing adjustments); in Java it's `System.nanoTime()`; most language runtimes expose an equivalent. These clocks are guaranteed to be non-decreasing for the lifetime of the process reading them, which is exactly the guarantee an elapsed-time measurement needs. The trade-off: - A monotonic clock's absolute value is meaningless outside the process/machine that produced it - you cannot compare `System.nanoTime()` values across two different JVMs, let alone two different machines. - The clock typically resets on reboot and may or may not continue advancing during system sleep depending on the platform. This makes monotonic clocks unsuitable for anything that needs to correlate with a human-readable date or with events on other machines - that job still belongs to the wall clock. ## The production failure modes The production failure modes from getting this backward are common and often subtle. 1. **Timeout and retry logic** implemented with wall-clock subtraction can compute a negative or near-zero "time remaining," causing either premature timeout firing or, worse, a timeout that never fires because the negative delta is treated as "plenty of time left," silently defeating the timeout's purpose. 2. **Duration-based metrics** (request latency histograms, job-runtime alerting) computed from wall-clock deltas can show impossible negative values or occasional huge spurious spikes that correlate suspiciously with known NTP correction windows or VM live-migration events - a classic on-call debugging trail is "the p99 latency graph shows a few-second spike at exactly the moment ops ran a fleet-wide clock resync." 3. **Rate-limiting or backoff algorithms** that compute "time since last attempt" via wall clock can misbehave the same way. 4. **Scheduled-task frameworks** that compute "time until next run" from wall-clock deltas can fire early, late, or in a tight loop if the wall clock jumps. ## The convention that fixes it The fix is a firm convention, not a one-off patch: - Any code measuring an interval or duration - benchmarking, timeouts, rate limiting, backoff, scheduling relative delays - should use the monotonic clock API. - Any code that needs to record, display, or compare an absolute point in time - log timestamps, "created_at" database columns, cross-service event correlation, expiry dates tied to a real calendar deadline - should use the wall clock. A useful worked example: measuring an HTTP request's latency for a metrics dashboard should be `nanoTime_end - nanoTime_start` (monotonic, immune to clock corrections), while the timestamp attached to the log line describing that request should be `currentTimeMillis()` (wall clock, meaningful to a human reading logs at a specific real time). Many production incident postmortems trace back to exactly this API being chosen for the wrong purpose.