Give me a repeatable, signal-driven decision procedure for isolating the bottleneck behind a Kafka p99/p999 latency spike across GC, page cache, slow followers, ISR shrink, request-queue saturation, network, hot partitions, and disk stalls.

## The principle: localize before you drill Tail-latency firefighting goes wrong when people jump straight to tuning GC or adding threads. The disciplined approach is a **funnel from broad to specific**, driven by Kafka's own additive timing signals, so every step narrows the search space with evidence. ## Step 0 — Define the spike Tail latency (p99/p999) is a **per-broker, per-request-type** phenomenon. First answer: which broker(s) and which request type (Produce, FetchConsumer, FetchFollower)? Compare per-broker percentiles; a single outlier broker reframes the whole investigation (likely hot partition or one degraded node). ## Step 1 — Phase split (the master cut) For the offending request type pull the 99th/999th percentile of each **RequestMetrics** phase: `RequestQueueTimeMs`, `LocalTimeMs`, `RemoteTimeMs`, `ThrottleTimeMs`, `ResponseQueueTimeMs`, `ResponseSendTimeMs` (they sum to `TotalTimeMs`). The dominant phase selects the branch. This is the single highest-leverage step. ## Step 2 — Branch by dominant phase **RemoteTimeMs dominant** → waiting on others. - Produce + acks=all → slow follower / replication. Check `IsrShrinksPerSec`, `UnderReplicatedPartitions`, per-replica lag; pin the lagging follower via its GC/disk/network/idle metrics. Remember ISR shrink can *mask* the latency while silently dropping durability (watch `min.insync.replicas`). - FetchConsumer → often *expected* long-poll purgatory (`fetch.max.wait.ms`); only pathological if beyond intent. **LocalTimeMs dominant** → leader-local stall. Overlay three timelines on the spike: - GC pauses (GC log / GarbageCollector MXBeans) — STW pauses freezing all threads, often regular cadence. - Page-cache misses — rising disk **read** IOPS/await, lagging/backfilling consumers (Kafka relies on OS page cache, not an internal cache). - fsync/flush stalls — rising disk **write** await, correlates with flush intervals/volume. **RequestQueueTimeMs dominant** → request-handler pool saturation. Read `RequestHandlerAvgIdlePercent` (toward 0 = saturated). Decide: genuinely under-provisioned (raise `num.io.threads`) vs threads *blocked* on GC/disk (adding threads won't help — fix the stall). Falling network idle + queueing on responses → `num.network.threads` / network. **ResponseSendTimeMs dominant** → network egress saturation or a slow/backpressured client TCP connection; check NIC utilization, large responses, and per-connection behavior. ## Step 3 — Cross-cut for skew Independently compare per-broker `BytesIn/Out/MessagesInPerSec` and per-partition rates to catch a **hot partition** concentrating load on one leader (key skew via `murmur2(key) % N`), which can produce a single-broker spike that the phase split alone won't explain. ## Step 4 — Confirm by closing the loop A hypothesis is only validated when the intended fix moves the **specific** phase/idle metric it should and p99 recovers. E.g. fixing a slow follower should drop Produce RemoteTime; adding RAM should raise the page-cache hit ratio and drop LocalTime. ## Why this ordering The phase metrics are additive and cheap, so they're the most information-dense first cut. Idle percents are leading saturation indicators. Subsystem metrics (GC, iostat, NIC) are *confirmatory*, pulled only after a phase points there. This prevents the classic anti-pattern of tuning a subsystem that isn't the bottleneck. ## Edge cases the procedure must handle - **Masking**: ISR shrink hides RemoteTime; throttling hides as ThrottleTime not LocalTime — read every phase, not just the obvious one. - **Multiple causes**: GC can both stall LocalTime and cool the page cache; rule each in independently. - **Expected vs pathological RemoteTime** for long-poll fetches. - **Single bad broker** vs cluster-wide: always start per-broker so you don't average away the culprit.