skip to content

Topic Configuration and Administration

3 roadmaps5 questions

Creating and altering topics, and knowing which config wins between topic override, broker default, and dynamic update. A practical question, especially the risks of auto.create.topics.enable.

part ofApache Kafkaoverview, primer and where to startread it →
on this pageshow

explore

questions

5

How do you create a Kafka topic, and what do the partitions and replication.factor settings mean?

level: juniorimportance: must knowfreq 80%

answer

  1. partitions = parallelism + ordering unit
  2. RF = copies across brokers
  3. RF <= live broker count
  4. kafka-topics.sh --create
  5. NewTopic(name, parts, rf)

basics

~10 s

Use the kafka-topics.sh CLI or AdminClient. partitions sets how many parallel log slices the topic has; replication.factor sets how many brokers keep a copy of each partition for fault tolerance.

solid answer

~30 s

A topic is created with the kafka-topics.sh --create command or programmatically via AdminClient.createTopics() passing a NewTopic. Two core settings: partitions controls the unit of parallelism and ordering — more partitions means more consumers can read concurrently, but ordering is only guaranteed within a partition. replication.factor controls how many broker copies exist of each partition (1 leader + N-1 followers); an RF of 3 tolerates 2 broker failures. RF cannot exceed the number of live brokers. Example CLI: kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 6 --replication-factor 3. You can also pass per-topic config overrides at creation with --config retention.ms=604800000.

go deeper

for a junior

Know the CLI create command and that partitions = parallelism, RF = copies.

for a middle

Explain ISR, leader/follower, RF <= broker count, and per-key ordering tied to partition.

for a senior

Discuss trade-offs of partition count (rebalance cost, file handles) and RF vs min.insync.replicas durability interplay.

for a principal

Reason about capacity planning: partition counts per broker, replication overhead, and topic sprawl governance across a cluster.

## What a topic is A Kafka **topic** is a named, append-only log of records. Each topic is physically split into one or more **partitions** — independent ordered sequences of records, each stored as a set of segment files on disk. Records within a partition have a monotonically increasing **offset**; ordering is guaranteed only *within* a partition, never across partitions. ## partitions The `partitions` count is the topic's unit of parallelism. A consumer group can have at most one active consumer per partition, so 6 partitions means up to 6 consumers reading in parallel. The record key's hash (default partitioner) decides which partition a record lands in, so all records with the same key stay in one partition and keep their relative order. You can *increase* partitions later (`kafka-topics.sh --alter --partitions N`) but never decrease them, and increasing them re-shuffles which partition future keys hash to (breaking per-key ordering across the boundary). ## replication.factor Each partition has one **leader** replica and zero or more **follower** replicas, spread across different brokers. `replication.factor=3` means 3 copies total. Producers and consumers talk only to the leader; followers fetch from the leader to stay in sync (the **ISR**, in-sync replica set). If the leader broker dies, a follower in the ISR is promoted. RF=3 is the common production default — it survives losing 2 brokers. RF must be <= number of live brokers at creation time, or creation fails with InvalidReplicationFactorException. ## Creating a topic CLI: ``` kafka-topics.sh --bootstrap-server localhost:9092 \ --create --topic orders --partitions 6 --replication-factor 3 ``` Programmatically: ```java NewTopic t = new NewTopic("orders", 6, (short) 3); admin.createTopics(List.of(t)).all().get(); ``` ## Edge cases - If you omit `--partitions`/`--replication-factor`, the broker defaults `num.partitions` and `default.replication.factor` apply. - Creating a topic that already exists throws TopicExistsException. - You can pass per-topic overrides at creation: `--config min.insync.replicas=2 --config cleanup.policy=compact`.

  • Can you decrease the partition count of a topic?
    No. Partitions can only be increased. Decreasing would require deleting records and would break key-to-partition mapping, so Kafka disallows it.
  • What happens if you set replication.factor higher than the number of brokers?
    Topic creation fails with InvalidReplicationFactorException — RF cannot exceed the number of live brokers.

saying these in an interview costs you the question

  • Saying ordering is guaranteed across the whole topic (it's only per-partition)
  • Claiming partitions can be decreased
  • Confusing replication.factor with min.insync.replicas
  • Thinking consumers read from followers by default (they read from the leader)

context

resolving links…

open as a page

Explain per-topic config overrides versus broker defaults: how does precedence work for something like retention.ms?

level: middleimportance: must knowfreq 65%

basics

~10 s

Brokers have cluster-wide defaults (e.g. log.retention.ms). A topic can override the equivalent topic-level config (retention.ms). If a topic-level override exists it wins; otherwise the broker default applies.

open as a page

What do auto.create.topics.enable and num.partitions control, and why is auto-creation often disabled in production?

level: middleimportance: should knowfreq 60%

basics

~20 s

auto.create.topics.enable lets the broker auto-create a missing topic when a client produces to or fetches from it. num.partitions and default.replication.factor decide the new topic's shape. It's disabled in prod to prevent typo'd or unplanned topics.

open as a page

Walk through the describe/alter-config workflow with kafka-configs.sh and AdminClient, including incrementalAlterConfigs versus the legacy alterConfigs.

level: seniorimportance: should knowfreq 45%

basics

~10 s

Use kafka-configs.sh --describe to read a topic/broker's configs and their source, and --alter --add-config/--delete-config to change them. Programmatically, prefer AdminClient.incrementalAlterConfigs (SET/DELETE/APPEND ops) over the deprecated alterConfigs, which replaced the whole config set.

open as a page

A topic's effective config value can come from several sources. What is the full precedence order Kafka uses to resolve a dynamic config, and how would you diagnose an unexpected effective value?

level: principalimportance: should knowfreq 30%

basics

~20 s

Order, highest to lowest: per-topic dynamic override, then per-broker dynamic config, then cluster-wide dynamic default broker config, then static broker config (server.properties), then the built-in default. To diagnose, describe the topic and read each entry's source tag.

open as a page