In windowed stream processing, what is a watermark, and how does it let a system decide when a time window is 'done' despite events arriving out of order?

## The problem: events do not arrive in order Windowed aggregation needs a well-defined moment to say 'this window is finished, emit the result' - but in a distributed streaming system, events don't arrive in the order they were generated. - A mobile client might buffer events offline and send a batch hours late. - A network partition might delay one producer while others keep flowing. - Retries can reorder delivery. If a stream processor decided a window was 'done' purely based on its own wall-clock processing time reaching the window's end, it would produce results based on whatever happened to have arrived by then - inconsistent and non-reproducible, since the same window could close with different contents depending on network conditions on a given run. ## What a watermark is A **watermark** solves this by giving the processor an explicit, principled notion of **event-time progress** that is decoupled from wall-clock processing time. Mechanically, a watermark is a monotonically non-decreasing value, computed from the event timestamps actually observed in the stream so far, that asserts: 'I do not expect to see any more events with a timestamp earlier than this value.' - A common, simple implementation is 'max observed event timestamp minus a bounded **out-of-orderness slack**' (e.g., 10 seconds) - as new events arrive, the watermark advances to track the newest timestamp seen, minus that slack, so it always trails slightly behind the freshest data to give stragglers a chance to arrive. - The watermark itself is periodically emitted through the stream topology as a special marker, alongside regular data records, so every downstream operator - including windowed aggregations - can independently track how far event time has progressed. ## When a window closes A window is considered closed, and its result is emitted, once the watermark passes the window's end boundary - not when wall-clock time does. This decouples 'when do we finalize this window's answer' from 'how fast is the pipeline actually running,' making results reproducible based on the data's own timestamps rather than incidental processing speed. This is the entire reason watermarks exist: they give a distributed, out-of-order stream a deterministic, data-driven closing signal instead of an arbitrary wall-clock deadline. ## The trade-off The unavoidable trade-off is that a watermark is a **heuristic**, not a guarantee - it's a bet about how out-of-order the stream can get. | Larger slack | Smaller slack | |---|---| | Setting the out-of-orderness slack larger makes the watermark more conservative, which reduces how much data arrives 'late' relative to the watermark, but it directly increases end-to-end latency, since every window's result is delayed by at least that slack. | Setting the slack smaller gets results out faster but increases how often genuinely valid events arrive after their window has already been declared closed - true late data. | There is no watermark setting that eliminates late data entirely for a genuinely unbounded-lateness real-world stream; you're always choosing a point on the **latency-versus-completeness curve**. ## Handling late data Because late data is a structural certainty, not an edge case, production systems need an explicit policy for it. The three common choices are: 1. **Drop it silently** - simplest, but silently undercounts affected windows, dangerous for anything business-critical. 2. **Route it to a separate 'late events' side output** so it can be inspected, alerted on, or reprocessed through a separate batch correction path. 3. **Re-fire the window**, in frameworks that support it (Flink's allowed lateness, for example): keep the window's state around for a further grace period after the watermark first closes it, and if a late event arrives within that grace period, re-fire the window with an updated result - which means downstream sinks must be able to handle a value for a given window being emitted more than once, effectively a correction, not just a single final answer. ## Failure modes - **Treating the first emission as final.** The most common production failure mode is treating the first window emission as final without accounting for this, then being surprised when a dashboard or downstream aggregate 'changes' for a time bucket that already looked finished - which is actually the system correctly applying a late-arriving correction, not a bug, but it breaks any downstream consumer that assumed 'once emitted, immutable.' - **A slack tuned only for the common case.** The second common failure is picking a fixed watermark slack that's tuned for the common case but starves out a legitimate long-tail of stragglers - e.g., a mobile app that syncs after being offline for an hour will always be treated as 'too late,' permanently and silently excluded from any window-based metric, no matter how the slack is tuned, unless a separate reprocessing path exists for it. ## Where it shows up A concrete real-world example: a ride-sharing platform computing 'trips completed per 5-minute window' per city uses a bounded-out-of-orderness watermark with a 30-second slack, tuned from observed network delay distributions. During a driver app's brief connectivity blip, a handful of trip-completion events arrive 40 seconds late - past the watermark - and are routed to a late-events topic; a separate nightly batch job reconciles those late events into the previously emitted per-window totals stored in the analytics warehouse, rather than trying to force the real-time path to wait indefinitely for every possible straggler.