skip to content

A VPN appliance's syslog places a session seven minutes before the sign-in that authorised it - why?

level: middleimportance: should knowfreq 52%

answer

  1. effect before cause is impossible
  2. which clock wrote each field
  3. the older syslog header has no offset
  4. whole hours mean timezone, minutes mean drift
  5. measure it with an event both sources saw

basics

~20 s

Almost certainly the appliance's clock, not the order of events. BSD-style syslog carries no timezone and no year, so a drifting or locally set clock is copied straight into the SIEM and can place an effect before its cause.

solid answer

~50 s

An effect cannot precede its cause, so the ordering is a claim about clocks, not about the intrusion. Three candidates, in order of likelihood: the appliance is free-running with broken or absent time synchronisation; it emits local time with no offset and the parser assumed UTC; or a daylight-saving transition has shifted an hour of its records. Seven minutes points at drift rather than a timezone, because timezone errors land on whole or half hours. Measure it rather than guess: find an **anchor event** that two sources both recorded - one authentication visible in both the identity provider and the appliance - and take the difference. Repeat over several anchors: a constant offset is a configuration error, a growing one is a free-running clock. Then store the measured offset and a normalised UTC field alongside the raw timestamp, never in place of it, and get the appliance onto NTP and RFC 5424 with an explicit offset.

code

text · 11 lines
text
# VPN appliance, RFC 3164 syslog - no year, no timezone, device-local clock
Mar 14 02:41:07 vpn-gw-fra sslvpn: session-start user=k.doyle src=203.0.113.44 tunnel=0x4f1a ...

# Identity provider sign-in record - RFC 3339, explicit UTC, provider's clock
{
  "createdDateTime": "2026-03-14T02:48:19Z",
  "userPrincipalName": "[email protected]",
  "appDisplayName": "VPN Gateway",
  "status": "success",
  ...
}

go deeper

for a junior

Be ready to say that an effect cannot precede its cause, so the ordering is a statement about clocks, and to name the obvious suspects: a drifting device clock or a log format that carries no timezone.

for a middle

Explain the mechanics - the older syslog header has no year and no offset, so local time ships verbatim - and distinguish drift, which grows, from a timezone error, which is a constant whole-hour offset.

for a senior

Demonstrate the measurement: anchor events seen by two sources, an offset tracked over time, the raw field preserved alongside a normalised UTC one, and a durable fix at the device rather than a hard-coded correction.

for a principal

Own the consequence when you cannot fix the device: a per-source offset register with named owners, a standing item on the risk list, and a rule that every investigation touching that source states its skew rather than absorbing it.

## The ordering is a claim about clocks When the record of a VPN session appears seven minutes before the authentication that created that session, exactly one of two things is true: your understanding of causality is wrong, or the two timestamps come from clocks that disagree. In practice it is always the second. The useful reflex is to stop reading the sequence as a story and start treating each timestamp as *a claim made by a specific clock*. ## Why appliance syslog is the usual culprit BSD-style syslog (RFC 3164) has a header of the form `Mmm dd hh:mm:ss hostname tag:`. It carries **no year and no timezone or UTC offset**. Whatever the device's local clock says is what ships, and the receiving pipeline has to guess the rest. RFC 5424 fixed this - its timestamp is an RFC 3339 value with an explicit offset - but plenty of appliances still emit the older format. That leaves three failure modes, and they look different: - **Drift.** The device has no working time synchronisation, or its NTP server is unreachable through a firewall rule nobody has reviewed since the device was installed. A free-running clock wanders by seconds a day, so the offset *grows*. Seven minutes is a classic drift number. - **Timezone naivety.** The device emits local time, the parser assumed UTC (or the wrong zone). This offset is *constant* and lands on a whole or half hour - one hour, five and a half hours - not on seven minutes. - **Daylight saving.** A source logging local wall-clock time duplicates an hour when clocks go back: two genuinely different hours carry identical labels, and their relative order is ambiguous. In spring the mirror problem appears - an hour of labels that never existed. A search bounded by that local hour returns two disjoint sets, or none. One asymmetry is worth knowing: domain-joined Windows hosts rarely show minutes of skew, because Kerberos rejects authentication beyond a default five-minute clock skew, so drift breaks logon before it reaches your SIEM. Standalone appliances, Linux hosts outside the domain, and cloud or SaaS sources have no such enforcement. Which is precisely why the appliance and not the identity provider is the suspect here. ## Measuring the offset instead of guessing it You usually cannot log into the appliance during triage, and you do not need to. Use an **anchor event**: one occurrence that two sources independently recorded. - An authentication the identity provider and the appliance both logged. - A connection visible in both the proxy and network flow records. - An action you generate yourself, if the source is still live and you are allowed to touch it. The difference between the two timestamps for that single occurrence is the offset. Repeat it across several anchors spread over days and read the shape: a flat offset is configuration, a rising one is drift, a step of exactly an hour on a known Sunday morning is DST. A cheap systemic version of the same check: chart, per source, event time minus ingest time. Any source whose event times sit *after* its own arrival times has a clock running ahead of yours, because a record cannot describe something that happens after it has already reached your collector. ## What to do with the offset The temptation is to correct the stored timestamp so the sequence reads properly. Do not. Keep the source's raw field exactly as it arrived, and add: - a normalised UTC field, - the offset that was applied, - and ideally the date from which that offset was measured. This matters for three reasons. First, a correction is a claim, and a claim has to be reproducible by someone who doubts you. Second, offsets expire: the day somebody finally fixes NTP on that appliance, a hard-coded correction starts producing wrong times with no warning, whereas a recorded offset with a validity date does not. Third, if the record ever has to support a statement you must defend, the raw value is the evidence and your normalisation is analysis - you keep them separable. ## Fixing it properly Triage-time correction is a workaround. The durable fixes are boring: put the device on a reachable NTP source and monitor synchronisation rather than assuming it; move the source to RFC 5424 or a JSON format with an explicit offset; where the device only speaks local time, configure it to UTC so no DST transition can ever duplicate or erase an hour; and set the parser explicitly rather than letting it default. Where you own none of those - a managed appliance, a provider co-running the estate - the offset register is the standing mitigation, and the skew belongs on a risk list with a named owner, because every investigation that touches that source pays for it again. ## What you can still say Even unfixed, the record is not worthless. It still proves the appliance observed a session, with a source address, a user and a duration. What it cannot do is order that session against another source's events to the second. Say that plainly rather than quietly picking whichever ordering supports the theory you already have.

  • How do you measure a source's offset when you cannot log into the appliance?
    Use an anchor event that two sources independently recorded - one authentication logged by both the identity provider and the appliance, or one connection seen in both proxy and flow data. The gap between the two timestamps for that single occurrence is the offset. Repeat across several anchors over several days: a flat offset is a configuration error, a growing one is a free-running clock.
  • What breaks in your data when a source logs local wall-clock time and the clocks go back for daylight saving?
    That source duplicates an hour: two genuinely different hours carry identical labels and their relative order inside the SIEM is ambiguous. In spring the mirror problem appears, an hour of labels that never existed. A search bounded by that local hour returns two disjoint sets or none, which is why sources should emit UTC or an explicit offset.
  • Should you correct the skewed timestamp in the stored record?
    No. Keep the raw field exactly as it arrived and add a normalised UTC value plus the offset you applied and the date you measured it. Silently rewriting the original makes your correction unreproducible, and it fails silently the day somebody fixes NTP on that device and the hard-coded offset becomes wrong.
  • The offset is exactly one hour and never changes. What does that tell you?
    That it is a timezone or parser assumption, not drift - free-running clocks do not land on round hours and stay there. Either the device emits local time and the pipeline read it as UTC, or the sourcetype is configured for the wrong zone. Fix it in the parser and at the device, and check whether the offset changes on the next daylight-saving transition, which confirms it is wall-clock local time.

saying these in an interview costs you the question

  • Concludes the session really did precede the authentication
  • Assumes every source ships UTC
  • Rewrites the raw timestamp so the sequence reads correctly
  • Blames the SIEM's search rather than the source clock
  • Thinks NTP on the domain controllers covers appliances too
  • Calls a seven-minute gap a timezone error

context

resolving links…