Topics, Partitions and Log Storage

Compaction keeps at least the latest value for each message key, deleting older values for that key. The default delete policy instead drops whole segments once they exceed retention.ms or retention.bytes, regardless of key.

An offset is a number that marks a record's position inside one partition. It starts at 0 and increases by 1 for each new record. Offsets are per-partition, so offset 5 in partition 0 and offset 5 in partition 1 are unrelated records.

Kafka guarantees order only within a single partition: messages are read in the same order they were written. It does NOT guarantee any order across different partitions of a topic.

It can only INCREASE the partition count of an existing topic, never decrease it. To shrink, you must create a new topic with fewer partitions and republish the data; Kafka has no in-place shrink.

Partition count is the parallelism ceiling for one consumer group. Each partition is assigned to at most one consumer in the group, so if a topic has 6 partitions, at most 6 consumers do useful work; extra consumers sit idle.

A record batch is a group of messages stored together as one unit on disk. Kafka batches records to reduce per-message overhead, improve compression, and write/read more efficiently — fewer, larger I/O operations instead of many tiny ones.

cleanup.policy=delete means old log data is discarded once it exceeds a time or size limit. Kafka deletes whole log segments older than the retention time, or to keep the partition under the size cap.

Use the kafka-topics.sh CLI or AdminClient. partitions sets how many parallel log slices the topic has; replication.factor sets how many brokers keep a copy of each partition for fault tolerance.

A topic is a named stream of records (like a category or feed name). Each topic is split into one or more partitions. A partition is an ordered, append-only log of records that Kafka stores and serves.

With a null key there's no hash to route on, so the producer spreads records across partitions. Modern Kafka uses sticky partitioning: it fills one partition's batch, then switches to another, instead of strict round-robin per record.

A tombstone is a record with a null value that marks a key for deletion. After compaction, the tombstone removes prior values for that key, and the tombstone itself is retained for at least delete.retention.ms before being purged, so lagging consumers can still observe the delete.

The current position is the offset of the next record the consumer will fetch — it lives in memory and moves forward as you poll. The committed offset is the position durably saved to Kafka, used to resume after a restart or rebalance. They can differ.

Records with the same key are hashed to the same partition by the producer's default partitioner. Since one partition is strictly ordered, all records for a given key are processed in the order they were produced.

CreateTime is the timestamp the producer set when the message was created. LogAppendTime is when the broker appended it. The topic config message.timestamp.type (CreateTime or LogAppendTime) decides which one Kafka keeps.

Time retention (log.retention.ms/.minutes/.hours, default 7 days) removes data older than a duration. Size retention (retention.bytes, default -1 = unlimited) caps bytes per partition. When both are set, a segment is removed if either limit is exceeded.

Brokers have cluster-wide defaults (e.g. log.retention.ms). A topic can override the equivalent topic-level config (retention.ms). If a topic-level override exists it wins; otherwise the broker default applies.

Because Kafka stores, replicates, and serves data per partition. Each partition lives on a broker, has its own replicas, and can be consumed by exactly one consumer in a group at a time — so partitions, not topics, drive parallelism.

The high watermark is the offset up to which records are replicated to all in-sync replicas, so they are 'committed' and safe. Consumers can only read records below the high watermark, never up to the LEO, because un-replicated records could be lost on leader failure.

With multiple in-flight batches and retries, a retried earlier batch can land after a later one, reordering within a partition. Enabling the idempotent producer preserves order even with up to 5 in-flight requests.

The default partitioner routes a key via hash(key) % numPartitions. Changing numPartitions changes the modulo result, so a key that always landed in partition 2 may now land elsewhere — splitting that key's records across partitions and breaking per-key ordering.

Each batch header carries a producerId (PID), producerEpoch, and baseSequence number. The broker tracks the last sequence it accepted per (PID, partition). If a retried batch repeats sequences, the broker drops it as a duplicate, so retries don't create duplicate records.

If a record has a key, the default partitioner hashes the key (murmur2 mod partition count) so the same key always maps to the same partition, preserving per-key order. Keyless records are spread across partitions (sticky/round-robin), so their relative order across partitions isn't guaranteed.

Use a ProducerRecord constructor that takes a partition number, e.g. new ProducerRecord(topic, partition, key, value). Then the partitioner is skipped entirely and the record goes to exactly that partition. You become responsible for balancing and for valid partition numbers.

Log-start-offset is the offset of the earliest record still retained. LEO (log-end-offset) is the offset that will be assigned to the next record — one past the last written record. Base offset is the first offset of a particular log segment file.

The batch header stores one absolute baseOffset. Each record inside stores only a small offsetDelta. A record's absolute offset = baseOffset + offsetDelta. This avoids repeating the full 8-byte offset on every record.

Use kafka-configs.sh (or kafka-topics.sh --config) to set per-topic overrides like retention.ms and retention.bytes. A topic-level config always overrides the broker default (log.retention.*). Inspect with --describe.

auto.create.topics.enable lets the broker auto-create a missing topic when a client produces to or fetches from it. num.partitions and default.replication.factor decide the new topic's shape. It's disabled in prod to prevent typo'd or unplanned topics.

A TopicPartition is the identity of one specific partition: the pair (topic name, partition number). A single record is uniquely addressed by adding its offset, giving the triple (topic, partition, offset).

Implement org.apache.kafka.clients.producer.Partitioner when the default key-hash routing isn't what you need — e.g. routing hot keys specially or grouping by a field of the value. You override partition(...) to return the partition number and wire it with partitioner.class.