How do you read kafka-consumer-groups --describe output, and why is offset lag not the same as end-to-end latency?

## The CLI ``` kafka-consumer-groups.sh --bootstrap-server host:9092 --describe --group my-group ``` For every partition the group is responsible for, you get columns: - **GROUP** — the consumer group id. - **TOPIC** / **PARTITION** — the topic-partition. - **CURRENT-OFFSET** — the group's **committed** offset (from `__consumer_offsets`); the offset of the next record it intends to read. - **LOG-END-OFFSET** — the broker's **LEO** for that partition (next offset to be produced). - **LAG** — `LOG-END-OFFSET - CURRENT-OFFSET` (the message-count lag). - **CONSUMER-ID** — the member instance currently assigned the partition. - **HOST** / **CLIENT-ID** — where that member runs and its client id. Reading rules: - A `-` (dash) in **CURRENT-OFFSET / LAG** means the group has **no committed offset** for that partition yet (brand-new group, or it never reached that partition). - A `-` in **CONSUMER-ID / HOST** means **no active member owns** the partition right now (group is empty/stopped, or the partition is unassigned mid-rebalance). The CLI still prints CURRENT-OFFSET, LOG-END-OFFSET, and LAG from stored data, which is how you inspect a stopped group. - Useful variants: `--all-groups`, `--state`, `--members --verbose`, and `--reset-offsets` (for repositioning). ## Why offset lag ≠ end-to-end latency These are **different quantities with different units**: - **Offset lag** = a **count of messages** the consumer hasn't read (LEO - committed). - **End-to-end latency** = a **duration**: the wall-clock time from when a record was **produced/appended** (its log-append or create timestamp) until it is **consumed/processed**. The conversion between them depends on **throughput**: - 100,000 msg/s with offset lag 100,000 → roughly **1 second** of time latency. - 1 msg/s with offset lag 100 → roughly **100 seconds** of time latency. So identical offset lag can mean wildly different time delays. Conversely a small offset lag on a slow topic can hide a large staleness. ## Measuring true end-to-end latency Because offset lag can't be converted to time without knowing throughput, measure latency directly: - Use the record's **timestamp** (`ConsumerRecord.timestamp()` / `timestampType` — either `CreateTime` set by the producer or `LogAppendTime` set by the broker) and compare it to the **time you finish processing** the record: `e2eLatency = processingTime - record.timestamp()`. - Or embed an explicit produce timestamp in the payload/header for end-to-end (producer-to-final-sink) timing that survives transformations. - Watch for **clock skew** between producer and consumer hosts when using CreateTime; LogAppendTime uses the broker clock and avoids producer skew but only measures broker-to-consumer, not true producer-to-consumer. ## When to use which - **Offset lag** is cheap, always available (CLI/exporter), and great for trend/backlog and retention-risk alarms. - **End-to-end latency** is what SLAs are usually written in ("events processed within 2s"), so for latency SLOs you must measure time directly rather than infer it from lag. - Together: rising offset lag warns of a building backlog; the latency metric tells you whether that backlog already breaches the time SLA.