What are the records-lag and records-lag-max client metrics, and why might they differ from what kafka-consumer-groups reports?
answer
- consumer-fetch-manager-metrics JMX group
- records-lag per partition, records-lag-max = max
- client = read position; CLI = committed position
- client metric vanishes when consumer dies
- fast-read slow-commit -> divergence
basics
~20 srecords-lag is a per-partition consumer client (JMX) metric showing how far behind that fetch is; records-lag-max is the maximum across the consumer's assigned partitions. They come from the live client, so they only exist while the consumer is fetching and reflect what it has fetched, not committed.
solid answer
~50 srecords-lag and records-lag-max are client-side JMX metrics exposed by the Java KafkaConsumer under the consumer-fetch-manager-metrics group. records-lag is per topic-partition (lastFetchedOffset relative to the partition's log-end / high-water mark seen in the fetch response); records-lag-max is the max across all partitions the consumer currently owns. They are computed inside the running consumer from fetch responses, so they only exist while the consumer is alive and actively fetching, and they reflect the consumer's *read* position, not its *committed* position. By contrast kafka-consumer-groups --describe computes lag from the broker's view: LEO minus the offset stored in __consumer_offsets. So the CLI sees committed lag even for a dead consumer, while records-lag-max sees in-flight read lag and goes away when the consumer stops. If a consumer reads fast but commits slowly, records-lag-max can look healthy while CLI/__consumer_offsets lag looks large.
go deeper
Know records-lag-max is a per-consumer client metric showing how far behind it is, max across partitions.
Distinguish client read-position lag from broker committed lag and know the JMX group name consumer-fetch-manager-metrics.
Explain divergence scenarios (slow commit, dead consumer, rebalance) and why you need both client and external monitoring.
Architect layered lag observability: cheap immediate client metrics plus durable external committed-lag monitoring, and reason about which governs SLOs and restart reprocessing.
## The two metric families There are two fundamentally different vantage points for measuring lag: 1. **Client-side (in-process JMX metrics)** — emitted by the running `KafkaConsumer`. 2. **Broker/external view** — computed from `__consumer_offsets` (committed) vs. the broker's log-end offset, used by `kafka-consumer-groups --describe`, Burrow, kafka-exporter, etc. ### records-lag and records-lag-max (client-side) These live in the JMX metric group **`kafka.consumer:type=consumer-fetch-manager-metrics`**: - **records-lag** — a *per-partition* gauge (attribute name `<topic>-<partition>.records-lag`): how many records behind the latest available offset this consumer is for that partition, derived from the most recent fetch response (the consumer's fetch position vs. the high-water mark the broker returned). - **records-lag-max** — the **maximum** records-lag across all partitions the consumer is currently assigned. This is the single number teams usually alert on for a consumer instance. - (There is also **records-lag-avg** for the average.) Key properties: - Computed **inside the JVM** of the running consumer from fetch responses — no extra broker calls. - Reflect the consumer's **read/fetch position**, i.e. how far its in-flight consumption is, *not* its committed offset. - Exist **only while the consumer is alive and fetching**. Kill the consumer and the metric disappears — you lose visibility exactly when you may need it most. - Reset when partition assignment changes (rebalance) because the per-partition attributes are keyed by assignment. ### kafka-consumer-groups --describe (broker view) This CLI reads the **committed** offset from `__consumer_offsets` and subtracts it from the broker's LEO. Therefore it: - Reports **committed lag**, which can lag behind read lag if commits are infrequent or `enable.auto.commit=false` with delayed manual commits. - Works even when **no consumer is running** (shows the standing committed position and current LEO; the CONSUMER-ID/HOST columns show '-'). - Is the basis for most external monitoring (Burrow, kafka-exporter). ## Why they differ | Aspect | records-lag-max (client) | kafka-consumer-groups (broker) | |---|---|---| | Source | running consumer JVM | __consumer_offsets + LEO | | Measures | read/fetch position | committed position | | Available when consumer dead? | No | Yes | | Granularity | per assigned partition / max | per partition in group | Concrete mismatch scenarios: - **Fast read, slow commit**: a consumer that reads aggressively but commits every 30s will show low records-lag-max but high CLI lag between commits. - **Dead consumer**: records-lag-max is gone; CLI still shows the frozen committed lag growing as producers add data — this is how you detect a crashed consumer externally. - **Rebalance churn**: records-lag per-partition attributes appear/disappear as partitions move, which can create gaps or spikes in dashboards if not handled. ## Practical guidance Use **records-lag-max** for fine-grained, low-latency, per-instance health (it is cheap and immediate), but back it with an **external** committed-lag monitor (kafka-exporter / Burrow) so you retain visibility when consumers crash and so you measure the durable committed progress that actually governs reprocessing on restart.
- Why is relying solely on records-lag-max dangerous for production alerting?Because it is emitted only by the live consumer JVM. If the consumer crashes the metric disappears, so you can lose lag visibility precisely when lag is exploding. You need an external committed-lag monitor (kafka-exporter, Burrow) as the durable safety net.
- Which JMX metric group exposes records-lag-max, and what's the difference between records-lag-max and records-lag-avg?It is under kafka.consumer type=consumer-fetch-manager-metrics. records-lag-max is the maximum lag across assigned partitions (worst case), records-lag-avg is the average across them — max is better for catching a single stuck partition.
saying these in an interview costs you the question
- Saying records-lag-max comes from __consumer_offsets — it is computed client-side from fetch responses.
- Claiming records-lag-max is still available after the consumer process dies.
- Conflating read-position lag (client) with committed-position lag (CLI/broker).
- Thinking records-lag-max is a broker-side MBean (it is a consumer-client metric).