Topics, Partitions and Log Storage

The log cleaner splits each partition into a 'clean' head (already compacted) and a 'dirty' tail (new records). It computes the dirty ratio = dirty bytes / total bytes and only cleans a partition when that ratio exceeds min.cleanable.dirty.ratio (default 0.5). The dirtiest eligible log is cleaned first.

min.compaction.lag.ms sets a minimum age a record must reach before the cleaner may compact it away. It guarantees that every version of a key stays in the log for at least that long, so consumers reading within that window can observe intermediate updates, not just the latest value.

Kafka can look up, per partition, the offset of the earliest record whose timestamp is >= a given time, using the consumer's offsetsForTimes (ListOffsets API) and the per-segment .timeindex files. It returns null if no record has a timestamp at or after that time.

Kafka only delivers a partition in order; ordering breaks if the consumer hands records to a thread pool, processes multiple partitions concurrently per record-key, retries/dead-letters async, or commits offsets in a way that lets reprocessing reorder side effects.

Each partition costs open file handles (log segments + index files), broker memory, replication threads, and controller/metadata load. More partitions also slow leader-election failover and lengthen rebalances. Open file descriptor limits and end-to-end latency cap how many partitions a cluster can hold.

compression.type sets the codec (none, gzip, snappy, lz4, zstd) applied to the whole batch's record section, recorded in the batch header. The producer normally compresses; the broker keeps it as-is unless the topic's compression.type forces a different codec, which makes the broker recompress.

Each batch has one CRC-32C checksum covering the batch body (records + most of the header) to detect corruption. Because the whole compressed batch is a self-contained, opaque blob, the broker can stream it straight from the OS page cache to the network with sendfile — no decompression or JVM-heap copy.

Kafka deletes whole segment files, and only after a segment is closed (rolled) and fully past the limit. The active segment is never touched. segment.bytes and segment.ms control when segments roll, so they set how late data can actually be deleted versus its nominal retention.

If a consumer lags behind retention, Kafka can delete records before they're read, causing OffsetOutOfRange and data loss for that consumer. Mitigate by sizing retention above worst-case lag, monitoring lag, alerting, and using size limits as a disk safety valve.

Use kafka-configs.sh --describe to read a topic/broker's configs and their source, and --alter --add-config/--delete-config to change them. Programmatically, prefer AdminClient.incrementalAlterConfigs (SET/DELETE/APPEND ops) over the deprecated alterConfigs, which replaced the whole config set.

Each partition is an append-only log: records are added at the tail, each getting a monotonically increasing offset. The log-start offset is the earliest still-retained offset; the high watermark is the highest offset replicated to all in-sync replicas — consumers can only read up to it.

Keying by customer_id keeps each customer ordered but routes all of a hot customer's traffic to one partition, overloading it and its consumer. Options: composite keys to spread, a custom partitioner, more partitions, or accepting weaker per-key ordering — each trades ordering against balance.

compact,delete applies compaction (keep latest per key) AND time/size retention (drop old segments) on the same topic. Use it when you want current state per key but also need to bound the topic's age or size so unbounded keys or stale data eventually age out.

Offsets are independent per-partition counters, so they can't order events across partitions. Kafka gives no global ordering by design. Alternatives: use a single partition, key-partition only what must be ordered, or add an application-level timestamp/sequence and reorder downstream.

A single partition gives total topic order but caps throughput to one consumer and one broker. Prefer keyed multi-partition topics so you keep per-entity order while scaling, reserving single-partition for genuinely global-order, low-volume streams.

Create a new topic with the desired partition count/key scheme, then republish: a bridge consumer reads the old topic and a producer writes into the new one with the new partitioning. Migrate consumers, then producers, to the new topic, drain the old, and retire it.

Order, highest to lowest: per-topic dynamic override, then per-broker dynamic config, then cluster-wide dynamic default broker config, then static broker config (server.properties), then the built-in default. To diagnose, describe the topic and read each entry's source tag.