Operations and Administration

kafka-log-dirs.sh reports, per broker and per log directory, how much disk each partition replica uses. You use it to find disk hot spots, balance data across JBOD log.dirs, and confirm reassignment/throttling progress.

broker.rack tags each broker with a fault domain (e.g., an availability zone). Kafka then spreads a partition's replicas across different racks, so losing one rack/AZ doesn't take all copies. When scaling or decommissioning, keep replicas balanced across racks.

Kafka serves recent reads from the OS page cache (RAM) instead of disk, so consumers that keep up stay memory-fast. Give the JVM a modest heap (~5-6 GB) and leave most RAM free for page cache. For network, size NICs above peak replication + produce + fetch traffic, which RF multiplies.

Kafka resolves the most specific source first: per-broker dynamic, then cluster-wide dynamic default, then the static server.properties value, then the hardcoded default. Topic-level overrides win over the broker default for that topic.

Sensitive dynamic configs (e.g. passwords) are stored encrypted in the metadata store, not in plaintext. Each broker needs password.encoder.secret in server.properties to encrypt/decrypt them, and kafka-configs --describe shows such values as redacted/[hidden].

Alert if the active controller count isn't exactly 1 (0 = no controller, >1 = split brain), and on rising controller queue size or slow leader elections. In KRaft, also watch the metadata quorum: leader presence, follower lag, and unfetched metadata.

Confirm the alert and identify the down broker. Check if it's process-down vs disk-full vs network-isolated. If recoverable, restart/heal it and watch URP drop as replicas rejoin ISR. If data is lost, reassign replicas. Escalate if min.insync.replicas is breached or partitions go offline.

Cruise Control is a tool that continuously monitors a Kafka cluster's load and automatically generates and executes reassignment plans to keep it balanced. It optimizes against a prioritized list of 'goals' (hard and soft constraints) and can self-heal by reacting to broker failures or anomalies.

KRaft has no ZooKeeper, so there is no inter.broker.protocol.version to set in server.properties. Instead the cluster has a feature flag named metadata.version that gates new behavior. You roll new binaries first (metadata.version stays put), then explicitly raise it with kafka-features.sh upgrade once all nodes are on the new version.

Watch remote copy lag (bytes/segments waiting to upload), remote copy throughput, and upload/delete error rates, plus local disk usage. Rising copy lag means uploads can't keep up, so segments stay local and disk fills.

The RSM is the pluggable component that physically reads/writes log segments and indexes to the remote backend. To run one against S3 you configure its class name plus backend settings: bucket, region/endpoint, credentials, chunk/part size, and optional encryption.

Run a passive DR cluster in another region, replicate topics asynchronously with MirrorMaker 2 or Cluster Linking, replicate consumer offsets and ACLs/config, and keep clients pointed at the active cluster. On disaster, translate offsets, repoint clients to DR, and run only one active cluster at a time to avoid split-brain. Failback re-syncs the original direction.

Pick brokers so total disk/network/partition load fits with each broker under ~60-70% utilization, you have at least RF brokers (plus spare for min.insync.replicas), and the cluster keeps working when N brokers fail. Survivors must absorb the dead brokers' load and re-replication.

Page on a small set of symptom alerts tied to user-facing SLOs (availability, durability, freshness) — offline partitions, write failures, SLO-breaching lag. Route cause-based and predictive signals (disk filling, URP, rising controller queue) to tickets/dashboards, not pages, unless they breach an SLO.

Kafka adds the new target replicas and waits for them to catch up and join the ISR before removing the old replicas, so the partition always keeps enough in-sync copies. Leadership only moves to a new replica once it is in the ISR, so no acknowledged data is lost.

Model local disk from local.retention and ingest rate; model remote cost from total retention × ingest × replication-of-1 (remote stores one copy). Add hidden costs: API request charges (PUT/GET/LIST), egress on remote fetches, and the metadata topic.