skip to content

What are the phases of a Kafka broker request as exposed by RequestMetrics, and how do they sum into TotalTimeMs?

level: middleimportance: must knowfreq 70%

answer

  1. RequestQueue → Local → Remote → Throttle → ResponseQueue → ResponseSend
  2. phases sum to TotalTimeMs
  3. kafka.network:type=RequestMetrics
  4. Remote = purgatory/replication wait
  5. diagnose by which phase dominates

basics

~20 s

Kafka splits each request's time into phases: time waiting in the request queue, time processing locally, time waiting on other brokers, time in the response queue, and time sending the response. TotalTimeMs is their sum.

solid answer

~40 s

Each Kafka broker request is broken into ordered phases recorded by the RequestMetrics MBean (kafka.network:type=RequestMetrics, name=<phase>, request=<ApiKey>): RequestQueueTimeMs (waiting in the request queue for an I/O thread), LocalTimeMs (leader processing on the I/O thread, e.g. writing to the log), RemoteTimeMs (waiting on other brokers — replication acks or purgatory for fetch/produce), ThrottleTimeMs (quota throttling delay), ResponseQueueTimeMs (waiting in the response queue), and ResponseSendTimeMs (time to write the response back to the socket). TotalTimeMs is the end-to-end sum of all these phases. The key diagnostic value is that you compare phases to localize a latency problem: a large RemoteTimeMs points to slow replication or long-polling fetches, large queue times point to insufficient threads, and large LocalTimeMs points to slow disk/log processing.

go deeper

for a junior

Know that a request's time is split into phases and TotalTimeMs is their sum; you can name a couple (queue time, local time).

for a middle

Name all phases in order, know what each measures, and that they're additive into TotalTimeMs.

for a senior

Diagnose by identifying the dominant phase and mapping it to a root cause (threads/disk/replication/network/quota).

for a principal

Build dashboards/alerts on per-phase p99 by ApiKey and reason about capacity (thread pools, replication topology) from the breakdown.

## Background Kafka brokers expose detailed per-request timing through JMX so you can see *where* a request spends its time, not just how slow it is overall. These metrics live under the MBean `kafka.network:type=RequestMetrics,name=<MetricName>,request=<ApiKey>`, where ApiKey is e.g. `Produce`, `Fetch`, `FetchConsumer`, `FetchFollower`, `Metadata`. Each is a histogram (Mean, 99thPercentile, Max, etc.). ## The request lifecycle (in order) A request flows through the broker like an assembly line: 1. **RequestQueueTimeMs** — A network thread (`Processor`) reads the request off the socket and puts it on the shared **request queue**. This metric measures how long it sat there before a **request handler / I/O thread** (`KafkaRequestHandler`, sized by `num.io.threads`) picked it up. High values mean too few I/O threads or saturated handlers. 2. **LocalTimeMs** — Time the I/O thread spends doing the leader-local work: validating, and for a produce, appending to the local log (page cache / disk); for a fetch, reading from the log. High values point to slow disk, page-cache misses, or heavy compression/validation. 3. **RemoteTimeMs** — Time the request waits in **purgatory** for something external before it can complete. For `Produce` with `acks=all`, this is waiting for follower replicas to acknowledge. For `Fetch`, this is the request long-polling in purgatory until `fetch.min.bytes` of data is available or `fetch.max.wait.ms` elapses. A large RemoteTimeMs on fetch is often *normal* (consumers idling); on produce it signals slow replication. 4. **ThrottleTimeMs** — If the client exceeds a configured **quota** (produce/fetch byte-rate or request-rate quota), the broker delays the response by this much. The response is held, then released. This is intentional backpressure, not a fault. 5. **ResponseQueueTimeMs** — After processing, the response is queued for a network thread to send. Time spent waiting in the **response queue**. 6. **ResponseSendTimeMs** — Time to actually write the response bytes back to the client socket. High values suggest network saturation or slow/backpressured clients. **TotalTimeMs = RequestQueueTimeMs + LocalTimeMs + RemoteTimeMs + ThrottleTimeMs + ResponseQueueTimeMs + ResponseSendTimeMs** (end to end). ## Why this matters Because the phases are additive, you diagnose by subtraction: look at TotalTimeMs's 99th percentile, then find which phase dominates. This converts a vague "Kafka is slow" into a precise root cause (threads vs disk vs replication vs network vs quotas). ## Edge cases - For `FetchConsumer`, large RemoteTimeMs is usually benign long-polling, not a problem. - ThrottleTimeMs appears both here and is echoed to the client in the response so the client can self-throttle. - Queue sizes (`RequestQueueSize`, `ResponseQueueSize`) are separate gauges that corroborate high queue-time readings.

  • If TotalTimeMs is high and almost all of it is RemoteTimeMs on Produce requests, what would you suspect?
    With acks=all, RemoteTimeMs is the wait for follower replicas to acknowledge. High values point to slow/lagging followers, network issues between brokers, or under-replicated partitions — not a local disk or thread problem.
  • Where does ThrottleTimeMs fit and is it a fault?
    It's the delay the broker adds when a client exceeds its quota. It's intentional backpressure, not an error; the value is also returned to the client so it can self-throttle and avoid more aggressive muting.

saying these in an interview costs you the question

  • Saying TotalTimeMs is measured independently rather than being the sum of the phases.
  • Claiming RemoteTimeMs is local disk time — it is the wait on other brokers / purgatory.
  • Thinking high RemoteTimeMs on consumer fetches is always a problem (it's usually benign long-polling).

context

resolving links…