skip to content

How does Kafka Streams handle out-of-order and late records in windowed operations, and what role does the grace period play?

level: seniorimportance: should knowfreq 45%

answer

  1. out-of-order = in grace, still updates window
  2. late = after windowEnd + grace, dropped
  3. close when streamTime >= end + grace
  4. TimeWindows.ofSizeAndGrace(size, grace)
  5. suppress untilWindowCloses = one final result

basics

~20 s

Out-of-order records (timestamp below stream time but within the grace period) are still added to their window and update the result. Once stream time passes window end plus the grace period, the window closes and later-arriving records for it are dropped as 'late'.

solid answer

~50 s

Kafka Streams windows operate on event time, so records can arrive out of order relative to stream time (the max timestamp seen). For a window [start, end), a record belongs to it by its event timestamp regardless of arrival order. Streams keeps the window 'open' for an additional grace period after end: while stream time < end + grace, an out-of-order record falling in that window is accepted and re-aggregates the result (emitting an updated value, unless suppressed). Once stream time crosses end + grace, the window is closed; any record for it arriving afterward is 'late' and dropped (incrementing the dropped-records metric). You set this with TimeWindows.ofSizeAndGrace(size, grace) (the older until()/grace-via-retention APIs are deprecated). Suppressed.untilWindowCloses(...) holds emission until close so you get a single final result. The grace period is the explicit knob trading completeness (tolerate more lateness) against latency and state size.

go deeper

for a junior

Know records can arrive out of order and that very late ones get dropped.

for a middle

Explain the grace period and that out-of-order-but-in-grace records update the window result.

for a senior

Detail the close condition (streamTime >= end + grace), suppression for final results, and the latency/state trade-off.

for a principal

Size grace from the observed lateness distribution, reason about suppression buffer policies and stream-time hazards, and balance correctness vs latency vs state at scale.

## Out-of-order vs late — the key distinction - **Out-of-order**: a record whose event timestamp is *less than* current stream time but whose target window is **still open** (stream time hasn't passed `windowEnd + grace`). These are fully processed: the record is placed in its event-time window and the aggregate is recomputed. - **Late**: a record arriving after its window has **closed** (stream time > `windowEnd + grace`). These cannot be incorporated; Streams **drops** them and records it in metrics (`dropped-records-total`, historically `late-record-drop`). ## The grace period Every windowed operation has a **grace period**: how long after a window's end Streams keeps accepting out-of-order records for it. Stream time is the clock. A window `[start, end)` is **closed** when `streamTime >= end + grace`. Before that, late-arriving-but-in-grace records update the result. - Defined with `TimeWindows.ofSizeAndGrace(Duration size, Duration grace)` (and similar for `SlidingWindows`, `SessionWindows`). Older APIs used window **retention/`until()`** to imply grace and are deprecated/error-prone. - Default grace in newer APIs is **0** (or 24h in some legacy defaults) — be explicit. ## Emission semantics By default, windowed aggregations emit a **continuously updating** result: each contributing record (including out-of-order ones) produces an updated downstream record. To emit only the **final** result once, wrap with `suppress(Suppressed.untilWindowCloses(BufferConfig...))`, which buffers and emits when stream time crosses the close boundary. This converts a noisy update stream into one record per window but adds latency (you wait for close) and requires a buffer (memory or with bytes/records bound and a shutdown/eager-emit policy). ## Trade-offs - **Larger grace** → tolerate more out-of-order data, more complete/correct results → but later final emission (higher latency) and larger windowed **state** retained. - **Smaller grace** → quicker close, less state → but more records dropped as late under real-world skew. - Grace interacts with stream-time hazards: a future-dated record can jump stream time and prematurely close windows, turning otherwise-in-order records into late drops. ## Monitoring Watch `dropped-records` metrics; rising drops mean the grace is too small for your actual lateness distribution. Inspect the lateness distribution (event time vs ingestion time gap) to size grace. ## Edge cases - Stream-stream joins also have a join window + grace; symmetric reasoning applies. - Session windows merge on new in-grace records, which can re-open/merge sessions until close.

  • What is the difference between an out-of-order record and a late record?
    Out-of-order: timestamp below stream time but its window is still open (within grace) — it's accepted and updates the result. Late: arrives after the window closed (stream time > end + grace) — it's dropped and counted in dropped-records metrics.
  • How do you emit only one final result per window instead of a stream of updates?
    Wrap the windowed aggregation with suppress(Suppressed.untilWindowCloses(...)). It buffers and emits the final value when stream time crosses windowEnd + grace, at the cost of added latency and buffer memory.

saying these in an interview costs you the question

  • Saying out-of-order records are always dropped (only late ones, past grace, are).
  • Confusing grace period with window size.
  • Thinking windowed aggregations emit only a final result by default (they emit continuous updates unless suppressed).
  • Believing a closed window can still accept records.

context

resolving links…