A branch office's endpoint events arrive hours after their host timestamps — what does that break during an intrusion investigation?

## Two clocks on every record Every record a collection pipeline stores carries at least two timestamps, and confusing them is one of the most expensive mistakes in a SOC. - **Event time** — written on the host, when the thing happened. - **Ingest time** — stamped by the collector or index, when the record arrived. In a healthy pipeline these differ by seconds. Across a saturated branch link, a backlogged agent doing exponential backoff, a batching schedule, or a collector working through a queue, they can differ by hours, and the difference varies by source and by time of day. ## What breaks, in order of how badly it bites **1. Scheduled detections silently miss.** A rule that runs every five minutes over the previous five minutes *of event time* evaluates a window and closes it. Records that arrive three hours later fall into a window already scanned. They are stored and fully searchable, and the rule will never look at them again. The failure produces no error and no alert — and the absence of an alert is indistinguishable from a genuinely quiet estate, which is precisely why this defect survives for months. Two fixes. Key the rule's window to *ingest* time with a lookback long enough to cover the observed skew, which guarantees every record is evaluated exactly once but means an alert can arrive long after the behaviour. Or keep event-time windows and add a grace period, re-evaluating older windows once late data has settled. Either way, the choice is explicit rather than accidental. **2. Timelines built from mixed sources imply false ordering.** Reconstruction pulls from sources with different lag: identity provider sign-ins land in seconds, a branch endpoint's process telemetry lands hours later. If you build the timeline from what is currently *present* rather than from event time across sources you know to be complete, you can conclude that the cloud sign-in preceded the local process launch when in fact it followed it — and an ordering claim is exactly the kind of thing a responder is asked to defend later. Order events by event time, and record for each source how much of the window has actually arrived. **3. Live response works on a stale picture.** You decide to isolate a host at 10:00 on evidence that runs to 06:00 because that is all that has landed. The forty minutes before your decision are still in flight. They will arrive, and they may change the scope. Note also the direction of the claim: pulling a host stops the damage and simultaneously freezes what you can learn — but with a lagging forwarder, isolation can also strand records that were still queued locally and never shipped. ## Lag is not clock skew They look the same on a timeline and have different fixes. **Lag** means the host wrote a correct timestamp and the record travelled slowly; it is measurable as ingest minus event time and correctable by waiting or re-scanning. **Clock skew** means the host wrote the *wrong* time; no amount of waiting fixes it, and the record will be misplaced in every timeline forever. Distinguish them by comparing a host's event times against the collector's receipt times over a long period: a steady offset with tight spread suggests skew, a variable and load-correlated offset suggests lag. Time synchronisation is the fix for one; capacity or buffering is the fix for the other. ## Measuring it Compute `ingest_time - event_time` per record, then aggregate per source, per site and per host class. Track the median and the p95, not an average — lag distributions are long-tailed, and the tail is the part that breaks rules. Alert when a source's p95 crosses the lookback your detections are configured for, because that is the moment the detections start missing. Watch the forwarder's queue depth alongside it; a rising queue predicts the lag before the lag shows up in the index. ## What you tell the investigation When you report on a window covered by a lagging source, say what fraction of it had arrived at the time you drew your conclusions, and re-run the key searches after the backlog drains. `We saw nothing from that site in the hour` is not a finding until you can say the hour had actually arrived.