Walk me through the per-phase timing metrics in a Kafka broker's request lifecycle. Which ones do you read first to localize where p99 latency is being spent?

## The problem "p99 latency is high" is useless on its own — you must know *where in the request's journey* the time goes. Kafka instruments this for you. ## What a broker request is Clients (producers, consumers) and other brokers (followers fetching to replicate) all send **requests** to a broker over the network: ProduceRequest, FetchRequest, MetadataRequest, etc. The broker processes each one and sends a response. Kafka measures the wall-clock time each request spends in each internal stage. ## The thread pipeline (why the phases exist) 1. **Network threads** (`num.network.threads`) read bytes off the socket and place a parsed request onto a shared **request queue**. 2. **I/O / request-handler threads** (`num.io.threads`) pull from that queue and actually do the work (append to log, read from log). 3. Some requests cannot complete immediately and are parked in **purgatory** (a delayed-operation holding area) — e.g. a Produce with `acks=all` waits there until enough followers replicate; a Fetch with `fetch.min.bytes` waits until enough data accumulates or `fetch.max.wait.ms` elapses. 4. The response is queued, then a network thread writes it back to the socket. ## The additive phase metrics Under JMX `kafka.network:type=RequestMetrics,name=<Phase>,request=<Produce|FetchConsumer|FetchFollower|...>`, each phase is a histogram (Mean, 99thPercentile, 999thPercentile, Max): - **RequestQueueTimeMs** — time the request sat in the request queue before an I/O thread picked it up. High = not enough I/O threads, or I/O threads blocked. - **LocalTimeMs** — time the I/O thread spent doing leader-local work (validating, appending to / reading from the log, page-cache access). High = GC pauses, disk stalls, lock contention, compression/validation cost. - **RemoteTimeMs** — time the request spent **blocked waiting on something external**: replication acks from followers (Produce acks=all), or sitting in fetch purgatory waiting for data. This is *expected* to be non-zero for acks=all and long-poll fetches; it's a problem when it spikes beyond your config's intent. - **ThrottleTimeMs** — time added by quota throttling (client/user quotas). - **ResponseQueueTimeMs** — time the finished response waited for a network thread. - **ResponseSendTimeMs** — time spent actually writing the response to the socket. High = network saturation or a slow/backpressured client TCP connection. - **TotalTimeMs** = sum of all of the above. Because they sum to the total, **comparing the p99 of each phase tells you which stage owns the latency.** That is the first cut of every tail-latency investigation. ## Worked example Produce p99 = 250 ms. You pull phases: RequestQueueTime p99 = 5 ms, LocalTime p99 = 8 ms, **RemoteTime p99 = 230 ms**, send = 3 ms. Conclusion: the leader processes fast; the time is spent waiting for follower replication (acks=all) — investigate slow/lagging followers or ISR, not broker CPU. Counter-example: RemoteTime low but **RequestQueueTime p99 = 200 ms** → I/O threads can't keep up; check `num.io.threads`, the request-handler idle percent, and whether I/O threads are blocked on GC or disk. ## Edge cases - RemoteTime being high for FetchConsumer is often *normal* (long-poll waiting for `fetch.min.bytes`). Don't treat expected purgatory waits as a bug — compare against `fetch.max.wait.ms`. - Mean can look fine while p99/p999 are terrible; always read the high percentiles, not the mean. - Percentiles are per-broker; a cluster-wide p99 spike on one broker often means one bad node.