Two servers in different data centers each timestamp an event using their local system clock (wall-clock/time-of-day). Why can't you safely assume that whichever timestamp is numerically smaller happened first in real time?

## Why a hardware clock is only approximate Every machine has a local hardware clock, typically a **quartz crystal oscillator**, that ticks at an approximately fixed frequency and drives a counter the OS turns into wall-clock time (time-of-day, e.g. milliseconds since the Unix epoch). "Approximately" is the key word: the oscillator's real frequency depends on manufacturing tolerance, temperature, supply voltage, and age, so no two machines' clocks tick at exactly the same rate. ## Drift and skew - **Clock drift** is that divergence in tick rate, usually measured in parts-per-million (ppm); a cheap commodity crystal can drift 10-100 ppm, which is up to several seconds per day if left uncorrected. - **Clock skew** is the accumulated difference between any two clocks at a given instant. Because every machine drifts independently and unpredictably, two clocks that agreed perfectly at some moment will disagree later. ## What synchronization does, and what it leaves behind To keep skew bounded, machines periodically synchronize against a reference, most commonly via **NTP**, which nudges the local clock back toward the reference. But synchronization is neither instantaneous nor continuous: - It happens periodically, seconds to minutes apart. - The correction itself carries its own uncertainty from network delay. - Between syncs, drift reaccumulates. Crucially, a correction can move the clock backward if it has drifted ahead of the reference - this is a **step** - or the synchronization daemon can **slew** the clock, temporarily running it faster or slower so it converges smoothly without ever going backward. Slewing is preferred but not universal: large corrections, cold boots, VM live-migration/resume-from-suspend, or manual admin changes can still produce a hard backward step. ## What that means for comparing two timestamps The practical consequence is that a timestamp read on machine A and one read on machine B are each best understood as "true time plus or minus some unknown, time-varying error," not an exact value. If A stamps event X at t=100ms and B stamps event Y at t=99ms, you cannot conclude Y happened first - the real ordering could easily be reversed, because each timestamp's error can be several milliseconds on a LAN and tens of milliseconds across a WAN, worse under congestion, virtualization jitter, or a misconfigured NTP daemon. Even on a single machine, wall-clock time is unsafe for measuring elapsed durations for the same reason a step correction can occur mid-measurement, which is why **monotonic clocks** exist as a separate, purpose-built tool. ## How it shows up in production This shows up in production in specific, painful ways. 1. A distributed store that resolves write conflicts by **"last write wins"** using client-supplied wall-clock timestamps can silently keep the wrong write and lose data if the writer that looks "later" actually had a fast clock - this is a well-documented footgun in systems that offer timestamp-based conflict resolution, and operators are advised to run tight clock discipline and still treat near-simultaneous writes as genuinely ambiguous rather than deterministically resolvable. 2. **Distributed tracing** that stitches spans together purely by wall-clock timestamp can render a child span as starting before its parent, confusing whoever is debugging a latency spike. 3. **Fleet-wide log correlation** by timestamp alone becomes unreliable for reconstructing exact event order without additional correlation IDs or sequence numbers. ## The engineering answer The general engineering answer is: never use raw wall-clock comparison to establish causal or global ordering across machines unless you have an explicit, quantified error bound and actively design around it. Two mechanisms do that: - **Bounded-uncertainty clocks.** This is exactly the idea behind bounded-uncertainty systems like Google's TrueTime, which report an interval guaranteed to contain the true time rather than a single falsely-precise number, and then have downstream logic (like Spanner's commit-wait) actively wait out that uncertainty before trusting an ordering decision. - **Logical or causal ordering.** Where that hardware investment isn't available or needed, the standard alternative is a purely logical or causal ordering mechanism - Lamport clocks, vector clocks, or hybrid logical clocks - that establishes happened-before relationships from message passing rather than physical time agreement. In practice, a well-run fleet with disciplined NTP keeps skew to low single-digit milliseconds, which is perfectly fine for human-facing logs and dashboards, but is not on its own a sound foundation for correctness-critical distributed ordering decisions.