skip to content

Explain CreateTime vs LogAppendTime in Kafka. How does message.timestamp.type affect what is stored in a record batch?

level: middleimportance: must knowfreq 45%

answer

  1. CreateTime = producer time (default)
  2. LogAppendTime = broker append time
  3. config: message.timestamp.type
  4. header: timestampType bit + firstTimestamp + maxTimestamp
  5. before/after.max.ms guards clock skew

basics

~10 s

CreateTime is the timestamp the producer set when the message was created. LogAppendTime is when the broker appended it. The topic config message.timestamp.type (CreateTime or LogAppendTime) decides which one Kafka keeps.

solid answer

~50 s

Each Kafka record carries a timestamp, and the **timestamp type** is set per topic via **message.timestamp.type** (default `CreateTime`). With **CreateTime**, the timestamp is whatever the producer assigned (usually wall-clock at send, or an app-supplied value); the broker preserves it. With **LogAppendTime**, the broker overwrites every record's timestamp with the broker wall-clock at append, giving monotonic, broker-controlled times. The batch header encodes this with a **timestampType** bit, stores a **firstTimestamp** plus per-record **timestampDelta**s (CreateTime), and a **maxTimestamp** field. CreateTime is what consumers, log retention, and time-based index lookups (offsetsForTimes) usually rely on, but it can be out of order or skewed by bad client clocks. LogAppendTime removes clock skew and guarantees ordering but loses the original event time. With CreateTime there is also **message.timestamp.before/after.max.ms** (newer brokers) to reject records whose producer timestamp drifts too far from broker time.

go deeper

for a junior

Know CreateTime = producer set it, LogAppendTime = broker set it, and the topic config that chooses.

for a middle

Explain how each is stored and which drives retention and time lookups.

for a senior

Discuss clock-skew guards, ordering guarantees, and event-time vs ingestion-time implications for Streams.

for a principal

Reason about cross-team contracts (event-time semantics), retention surprises from maxTimestamp, and when to mandate LogAppendTime.

## Two kinds of time Every RecordBatch v2 record has a **timestamp** (int64, epoch millis). But there are two possible *meanings*, selected by the topic config **`message.timestamp.type`**: 1. **CreateTime** (default): the timestamp comes from the **producer**. By default the producer client stamps `System.currentTimeMillis()` at send time, but an application can supply its own (e.g. the real event time) via the `ProducerRecord(timestamp, ...)` constructor. The broker **preserves** it. 2. **LogAppendTime**: the **broker** overwrites the timestamp with its own wall-clock at the moment it appends the batch to the log. Producer-supplied timestamps are discarded. ## How it is stored in the batch The **batch header** holds: - A **timestampType** flag bit (0 = CreateTime, 1 = LogAppendTime). - **firstTimestamp** (int64): the timestamp of the first record. - **maxTimestamp** (int64): the largest timestamp in the batch (used for retention and the time index). Each **record** stores a **timestampDelta** (varint) so its timestamp = `firstTimestamp + timestampDelta`. Under LogAppendTime the broker normalizes these so all records reflect the append time. ## Why it matters - **Retention & deletion:** time-based retention (`retention.ms`) and time index lookups (`consumer.offsetsForTimes`, the `.timeindex` file) use these timestamps. With CreateTime, a record with a wildly wrong producer clock can be deleted too early or too late, or break time-lookup monotonicity. - **Stream processing:** Kafka Streams event-time windows typically use CreateTime (the real event time). LogAppendTime is ingestion time, not event time. - **Ordering:** LogAppendTime is monotonically non-decreasing along the log; CreateTime can be out of order (clients buffer, retry, or backfill). ## Guarding bad clocks Newer brokers add **`message.timestamp.before.max.ms`** and **`message.timestamp.after.max.ms`** (which replaced the single `message.timestamp.difference.max.ms`). Under CreateTime, a batch whose timestamp drifts beyond these bounds from broker time is **rejected** with `InvalidTimestampException`. Under LogAppendTime these bounds are irrelevant since the broker stamps the time itself. ## Edge cases - Switching a topic from CreateTime to LogAppendTime does not rewrite existing data — only new appends change. - A producer can set timestamp = -1 to mean "use default"; the client then fills wall-clock. - The **maxTimestamp** drives `retention.ms`, so a single far-future CreateTime record can keep a whole segment alive longer than expected.

  • Which timestamp type does time-based retention and offsetsForTimes use, and what's the risk with CreateTime?
    Both use the record timestamps (the batch maxTimestamp for retention). With CreateTime, a skewed or far-future producer clock can delete segments too early/late or break the monotonicity that time-index lookups expect.
  • How do you stop clients with bad clocks from polluting a CreateTime topic?
    Set message.timestamp.before.max.ms / message.timestamp.after.max.ms (older brokers: message.timestamp.difference.max.ms); batches drifting beyond the bound are rejected with InvalidTimestampException. Or switch the topic to LogAppendTime.

saying these in an interview costs you the question

  • Saying LogAppendTime keeps the producer's original event time (it overwrites it).
  • Claiming the default is LogAppendTime (default is CreateTime).
  • Thinking changing message.timestamp.type rewrites existing records.
  • Confusing per-record timestamp with the batch's maxTimestamp used for retention.

context

resolving links…