Which controller metrics track how far the metadata log has been applied, and how do you use lastAppliedRecordOffset to detect a lagging or stuck controller?

## Two layers: replicated vs applied KRaft metadata moves through two stages on every node: 1. **Replication** — the node fetches records of the `__cluster_metadata` log and appends them to its local log (tracked by LogEndOffset / Lag, visible in `kafka-metadata-quorum.sh describe --replication`). 2. **Apply** — the node feeds those records through its state machine to build the in-memory **metadata image** (topics, configs, ACLs). This is what the `lastApplied*` metrics measure. A node can be **replicating fine but applying slowly** (or stuck), so you need both views. ## The key metrics | Metric | Meaning | |---|---| | `lastAppliedRecordOffset` | Highest metadata-log offset the node has applied to its in-memory state. | | `lastAppliedRecordTimestamp` | The append timestamp recorded in that last-applied record. | | `lastAppliedRecordLagMs` | `now − lastAppliedRecordTimestamp`: how stale the applied view is. | These are exposed by both **controllers** (`kafka.controller` / `kafka.server` MBeans) and **brokers** (which apply metadata as observers). The exact MBean names have shifted across Kafka versions, but the semantics above are stable. ## Detecting a lagging or stuck controller 1. **Compare across nodes.** Graph `lastAppliedRecordOffset` per node. On a healthy cluster all controllers track close together and the leader leads slightly. A follower whose offset **plateaus** while others advance is stuck applying — possible causes: slow/failing disk, GC pauses, a poison record, or a thread hang. 2. **Watch `lastAppliedRecordLagMs`.** A steadily **rising** lag means the node's metadata image is aging. On a broker, a high applied lag means it may route based on a stale topic/partition map — clients can see NOT_LEADER errors or miss new topics. 3. **Cross-check with quorum describe.** If `describe --replication` shows low Lag (replication healthy) but `lastAppliedRecordLagMs` is high, the bottleneck is in the **apply** path, not the network/fetch path. Conversely high Raft Lag points at replication/fetch. 4. **Snapshot interactions.** KRaft periodically writes **snapshots** of the metadata log so it doesn't grow unbounded. A node catching up from a snapshot will briefly show a jump in applied offset; don't mistake the snapshot-load gap for being stuck. ## Metadata error counts Controllers also expose error/anomaly counters (e.g. metadata loading or apply error counts) and `MetadataErrorCount`-style metrics. A non-zero and growing error count indicates the controller hit problems applying records — often correlated with an applied offset that stops advancing. Any non-zero value here warrants investigation because a controller that can't apply metadata diverges from the rest of the cluster. ## Edge cases - **Active controller vs followers**: the active controller applies records it just committed, so its applied offset is near the high watermark; followers apply after fetching, so a small steady gap is normal. - **Brokers are observers**: their applied lag matters for data-plane correctness (routing) even though they don't vote. - **Clock skew** can distort `lastAppliedRecordLagMs` because it uses record timestamps vs local now; large negative or wildly large values may indicate NTP problems rather than real lag.