What is the records-lag-max JMX metric, where does it come from, and what are its strengths and blind spots compared to broker-side lag?
answer
- Client-side consumer-fetch-manager-metrics MBean
- LEO arrives in fetch responses
- Max over assigned partitions
- Dead consumer reports nothing — broker lag still climbs
- Pair with external lag for absent-consumer detection
basics
~10 srecords-lag-max is a client-side JMX metric exposed by each consumer reporting the maximum lag across the partitions it currently fetches. It's cheap and real-time but only sees assigned partitions of running consumers.
solid answer
~40 srecords-lag-max is a consumer fetch metric (MBean kafka.consumer:type=consumer-fetch-manager-metrics,client-id=...) emitted by the Java consumer itself. For each fetch the consumer knows the partition's log-end-offset (returned in fetch responses) and its own position, so it computes per-partition records-lag and exposes records-lag-max as the worst case across assigned partitions; there are also per-partition records-lag and records-lag-avg. Its strength: it is in-process, near real-time, and needs no extra broker query. Its blind spots: it reports only partitions the consumer is *currently assigned and fetching*, so a crashed/dead consumer reports nothing (its lag silently grows on the broker side), and it can't see partitions that have no live owner. So records-lag-max is great for autoscaling and live dashboards of a running app, but external/broker-side lag (CLI or an exporter reading __consumer_offsets) is needed to catch stalled or absent consumers.
go deeper
Know it is a per-consumer JMX metric showing the worst partition's lag.
Explain it is client-side, derived from LEO in fetch responses, and is real-time but limited to assigned partitions.
Articulate the dead-consumer blind spot and why you must combine it with broker-side/external lag.
Design a layered lag-observability strategy (in-app metric for autoscaling + external exporter for absent-consumer alerts) and define the alert conditions.
## What it is `records-lag-max` is a **client-side** metric produced by Kafka's Java `KafkaConsumer`. It is not computed by the broker. The MBean is: ``` kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<client.id> ``` with attributes including `records-lag-max` (worst partition), and per-topic-partition `records-lag` / `records-lag-avg` under `...,topic=<t>,partition=<p>`. ## How the consumer knows the lag Every **fetch response** the broker sends back includes the partition's current **log-end-offset (LEO)** (more precisely the high watermark / last stable offset). The consumer also knows its own **fetch position** (the next offset it will request). So locally: ``` records-lag(partition) = partitionLEO - consumerPosition records-lag-max = max over currently-fetched partitions ``` Because this rides along on fetches the consumer is already doing, it is essentially free and updates continuously — no separate admin call to the group coordinator. ## Strengths - **Real-time** and low overhead; perfect for live dashboards and **lag-driven autoscaling** of a running consumer fleet. - Reflects the consumer's *actual fetch position*, which can be ahead of the *committed* offset (it sees in-flight progress, not just last commit). ## Blind spots (the key senior point) - **Only assigned, actively-fetching partitions count.** If a consumer **crashes, hangs, or the whole group goes down**, it stops emitting the metric — yet broker-side lag keeps climbing because producers keep writing. Relying solely on records-lag-max can make a dead consumer look like 'lag = 0 / no data', the most dangerous failure mode. - **No global view**: each instance reports only its slice. You must aggregate across instances, and you still miss partitions owned by no live member. - It tracks **fetch position**, not **committed** offset, so after a crash the *committed* position (what a new owner resumes from) may be behind what records-lag-max last showed. ## Broker/external lag complements it To cover the blind spots, pair it with **broker-side / external lag**: the CLI `--describe`, or an exporter (Burrow, Kafka Lag Exporter) that reads committed offsets from `__consumer_offsets` and partition LEOs via the admin protocol. Those see *all* partitions and *all* groups regardless of whether any consumer is alive — at the cost of not knowing in-flight (uncommitted) progress and adding polling overhead. ## Practical takeaway - Use **records-lag-max** for fast, in-app signals and autoscaling triggers. - Use **external lag** for alerting on stalled/absent consumers and SLO reporting. - Best practice: alert on *both* — high records-lag-max *or* (high broker-side lag AND no live consumer).
- A consumer group crashes entirely. What does records-lag-max show, and why is that dangerous?It shows nothing / stops updating, because no consumer is fetching. Broker-side lag keeps growing. Alerting only on records-lag-max would miss the outage — you need external lag to detect absent consumers.
- Why might records-lag-max differ from lag computed from committed offsets?records-lag-max uses the consumer's live fetch position, which can be ahead of the last committed offset. Broker-side lag uses the committed offset. The gap is in-flight, not-yet-committed work.
saying these in an interview costs you the question
- Claiming records-lag-max is computed by the broker (it is client-side).
- Believing it covers all partitions/groups — it only covers the consumer's currently assigned ones.
- Alerting solely on records-lag-max, which goes silent exactly when a consumer dies.
- Confusing fetch position (what this metric uses) with committed offset.