What is kafka-log-dirs used for, and how does it help with disk capacity and partition placement?
answer
- per-broker, per-log.dir replica SIZE in JSON
- JBOD multi-disk visibility
- isFuture = intra-broker move in progress
- offsetLag = move catching up
- error field = offline/failed disk
basics
~10 skafka-log-dirs.sh reports, per broker and per log directory, how much disk each partition replica uses. You use it to find disk hot spots, balance data across JBOD log.dirs, and confirm reassignment/throttling progress.
solid answer
~40 skafka-log-dirs.sh --bootstrap-server host:9092 --describe --broker-list 1,2,3 --topic-list orders returns JSON listing every broker's log directories (the paths in the log.dirs broker config) and, for each, the partition replicas it holds with their on-disk size in bytes and any offsetLag for replicas still moving between directories. It's the tool for disk-level visibility: which broker/disk is filling up, how big each partition replica is, and whether an intra-broker move (across JBOD disks) or a reassignment is progressing. Combined with the reassignment tools you can rebalance hot disks. The output is machine-readable JSON, so it's commonly parsed in scripts to alert on imbalance. It complements kafka-topics --describe (which shows replica *placement* across brokers) by adding the *size* dimension within each broker's disks.
go deeper
Know it reports disk usage per partition/broker as JSON.
Read size/offsetLag/isFuture and relate them to capacity and moves.
Use it to find disk skew and track intra-broker/cross-broker reassignments.
Build capacity/imbalance alerting and JBOD rebalancing policy from its JSON output.
## The gap it fills `kafka-topics --describe` tells you **where** replicas live (which brokers) but not **how big** they are or **which physical disk** holds them. On a broker configured with multiple data directories (**JBOD**, `log.dirs=/data1,/data2,...`), each partition replica lives in exactly one of those directories. `kafka-log-dirs.sh` exposes that size-and-placement detail. ## Command ``` kafka-log-dirs.sh --bootstrap-server broker:9092 \ --describe \ --broker-list 1,2,3 \ --topic-list orders,payments ``` Output is **JSON**, roughly: ```json { "brokers": [ { "broker": 1, "logDirs": [ { "logDir": "/data1", "error": null, "partitions": [ { "partition": "orders-0", "size": 10485760, "offsetLag": 0, "isFuture": false } ]} ]} ] } ``` Fields per partition: - **size** — bytes on disk for that replica (the dominant capacity signal). - **offsetLag** — for a replica being moved/rebuilt, how far behind it is. - **isFuture** — true while a replica is being relocated to a *different log dir on the same broker* (an intra-broker move); the "future" copy is built alongside the current one, then swapped. - **error** — non-null if a log dir is offline (e.g. a failed disk), which is itself a critical signal. ## What you do with it 1. **Find disk hot spots.** Sum sizes per logDir to see which physical disk on which broker is near full. Kafka does not automatically balance across JBOD disks on size, so skew is common. 2. **Plan/track moves.** After triggering an **intra-broker replica move** (reassigning a replica to a different log dir) or a cross-broker **partition reassignment** (`kafka-reassign-partitions.sh`), poll log-dirs to watch `offsetLag` shrink and `isFuture` clear. 3. **Detect failed disks.** A `logDir` with a non-null `error` indicates an offline directory; with JBOD a single disk can fail without taking the whole broker down, but its partitions go offline. 4. **Capacity alerting.** Because output is JSON, ops scripts parse it to alert on per-disk imbalance or near-full directories before they cause `NotEnoughReplicas`/produce failures. ## Relationship to other tools - `kafka-topics --describe` → replica **placement** across brokers + ISR health. - `kafka-log-dirs` → replica **size** and which **disk** within a broker. - `kafka-reassign-partitions` → actually **move** replicas (across brokers or, via log-dir targets, across disks) with throttling. Together they let you balance both broker-level and disk-level load. ## Edge cases - Sizes are per-replica, so a partition with RF=3 is counted three times across brokers — don't double-count when estimating cluster footprint vs. per-broker footprint. - During a move you may briefly see both the current and `isFuture` copy, inflating apparent usage until the swap completes. - A broker that is down won't report; missing brokers in the output are themselves a signal.
- kafka-topics --describe already shows replicas. Why also use kafka-log-dirs?kafka-topics shows placement (which brokers) and ISR health but not size or which physical disk holds each replica. kafka-log-dirs adds per-replica byte size and per-log.dir placement, which is essential for disk capacity planning and balancing JBOD disks.
- What do isFuture and offsetLag indicate in the output?isFuture marks a replica being relocated to a different log directory on the same broker — the new copy is built in parallel before the swap. offsetLag is how far that (or any rebuilding) replica is behind the source; it should trend to zero as the move completes.
saying these in an interview costs you the question
- Thinking kafka-log-dirs moves data (it only reports; reassign-partitions moves it).
- Assuming Kafka auto-balances data across JBOD disks by size.
- Double-counting replica sizes when estimating cluster footprint.
- Ignoring the error field that signals an offline log directory.