What is request purgatory in Kafka, and how does it relate to RemoteTimeMs for Fetch and Produce requests?

## What problem purgatory solves Many Kafka requests cannot be answered synchronously. Two big examples: - A **Produce with `acks=all`** must wait until all in-sync replicas (ISR) have replicated the records before the broker can acknowledge. - A **Fetch** with `fetch.min.bytes > 1` should wait (long-poll) until at least that many bytes are available, or until `fetch.max.wait.ms` elapses — so consumers aren't hammered with empty responses. If the broker held an I/O thread blocked for each such wait, it would run out of threads instantly. **Purgatory** decouples the wait from the thread: the request handler completes its synchronous work, then *parks* the request in a purgatory (a `DelayedOperationPurgatory`) and frees the thread. The request is completed later — either when its condition is satisfied (a follower's fetch advances the high-water mark; new data arrives) or when its timeout fires. ## Where the time shows up The wall-clock time a request spends parked in purgatory is recorded as **RemoteTimeMs** in RequestMetrics. "Remote" because the completion depends on something *other than* this broker's local processing — typically other brokers (replication) or future client activity (more produces filling a fetch). ## Two purgatories, two interpretations - **Produce purgatory** — high RemoteTimeMs here means ISR followers are slow to replicate: network issues, lagging brokers, or under-replicated partitions. This is a real latency problem. - **Fetch purgatory** — high RemoteTimeMs here is frequently *benign*: consumers configured with a fetch wait are simply long-polling an idle topic. Don't alert on it blindly. ## Observability - `kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Produce` (and `Fetch`) — number of requests currently parked. - `name=NumDelayedOperations` — similar gauge. There are also purgatories for other delayed operations (DeleteRecords, Heartbeat, Rebalance, etc.). ## Implementation detail Purgatory is built on a **hierarchical timing wheel**, an O(1) data structure for scheduling huge numbers of timeouts efficiently, plus a watcher-list keyed by the partitions a request waits on, so completion checks are triggered only by relevant events rather than scanning all parked requests. ## Edge cases - A request can complete *before* its timeout (condition met) or *at* timeout (forced completion returning whatever is available). - Large fetch purgatory size with low RemoteTimeMs simply means many idle long-polls — normal. - Producer with `acks=1` or `acks=0` skips produce-purgatory replication wait, so RemoteTimeMs stays near zero.