skip to content

How do you configure partitions, replicas, and topic-level settings (like a compacted topic) using TopicBuilder?

level: middleimportance: should knowfreq 35%

answer

  1. name/partitions/replicas required trio
  2. default 1 partition + 1 replica
  3. .config(TopicConfig.*_CONFIG, value)
  4. .compact() = cleanup.policy=compact
  5. assignReplicas XOR replicas

basics

~10 s

Use TopicBuilder.name("t").partitions(n).replicas(m), then add topic configs via .config(key, value) or .compact()/.configs(map). .build() returns a NewTopic bean that KafkaAdmin provisions.

solid answer

~30 s

TopicBuilder (org.springframework.kafka.config.TopicBuilder) fluently builds a NewTopic. name() is required; partitions(int) and replicas(short) set the layout — if you omit them TopicBuilder defaults to 1 partition and 1 replica, which is dangerous in prod. Topic-level configs (Kafka's per-topic properties keyed by TopicConfig constants) are set with .config(key, value), .configs(Map) or convenience methods: .compact() sets cleanup.policy=compact; there's also replica assignment control via .replicasAssignments(map) or .assignReplicas(partition, brokerList) for manual placement instead of a replication factor. Example: TopicBuilder.name("events").partitions(12).replicas(3).config(TopicConfig.RETENTION_MS_CONFIG, "604800000").compact().build(). Return it as a @Bean and KafkaAdmin applies partitions/replicas/configs at creation. Note configs only take effect at creation unless you enable modifyTopicConfigs for existing topics.

code

java · 18 lines
java
@Bean
NewTopic eventsTopic() {
    return TopicBuilder.name("events")
            .partitions(12)
            .replicas(3)
            .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7d, value is a String
            .config(TopicConfig.MIN_INSYNC_REPLICAS_CONFIG, "2")
            .build();
}

@Bean
NewTopic compactedTopic() {
    return TopicBuilder.name("user-snapshots")
            .partitions(3)
            .replicas(3)
            .compact()               // cleanup.policy = compact
            .build();
}

go deeper

for a junior

Know the name/partitions/replicas fluent chain and .build().

for a middle

Set topic configs via .config with TopicConfig keys, use .compact(), know the 1/1 default risk.

for a senior

Understand configs apply at creation only and the replicas-vs-assignReplicas exclusivity.

for a principal

Standardize partition/replica/min.insync.replicas policy across topics and environments.

## TopicBuilder API surface `TopicBuilder` produces a Kafka `NewTopic`. The important calls: - **`name(String)`** — required; the topic name (also usable as the static factory: `TopicBuilder.name("x")`). - **`partitions(int)`** — number of partitions (unit of parallelism/ordering). Default **1** if omitted. - **`replicas(short)`** — replication factor (copies across brokers for durability). Default **1** if omitted. Must be ≤ number of brokers or creation fails. - **`config(String key, String value)`** — one topic-level config. Keys come from `org.apache.kafka.common.config.TopicConfig`, e.g. `RETENTION_MS_CONFIG`, `CLEANUP_POLICY_CONFIG`, `MIN_INSYNC_REPLICAS_CONFIG`, `SEGMENT_BYTES_CONFIG`. - **`configs(Map<String,String>)`** — bulk set configs. - **`compact()`** — shortcut for `cleanup.policy=compact` (log compaction: keep the latest value per key). - **`replicasAssignments(Map<Integer,List<Integer>>)` / `assignReplicas(int partition, List<Integer> brokerIds)`** — manual replica placement per partition. **Mutually exclusive with `replicas()`** — you specify a replication factor *or* explicit assignments, not both. ## Topic-level configs vs client/admin configs Do not confuse **topic configs** (retention, cleanup.policy, min.insync.replicas — properties of the topic on the broker) with **producer/consumer/admin client configs** (bootstrap servers, serializers). TopicBuilder.config() sets the former. ## Example ```java @Bean NewTopic eventsTopic() { return TopicBuilder.name("events") .partitions(12) .replicas(3) .config(TopicConfig.RETENTION_MS_CONFIG, "604800000") // 7 days .config(TopicConfig.MIN_INSYNC_REPLICAS_CONFIG, "2") .build(); } @Bean NewTopic snapshotsTopic() { return TopicBuilder.name("snapshots") .partitions(3) .replicas(3) .compact() // cleanup.policy=compact .config(TopicConfig.MIN_COMPACTION_LAG_MS_CONFIG, "3600000") .build(); } ``` ## Gotchas - **Silent 1/1 default:** forgetting partitions/replicas yields a single-partition, single-replica topic — no parallelism, no durability. Always set both explicitly in shared environments. - **replicas > broker count** ⇒ creation fails with an error at startup (not fatal to the app unless you opt into fail-fast). - **Configs are applied at *creation* only.** If the topic already exists, TopicBuilder configs are ignored unless KafkaAdmin has `modifyTopicConfigs=true`. - **replicas() and assignReplicas() conflict** — pick one. - Config *values* are strings, even numeric ones (`"604800000"`, not a long).

  • What happens if you call replicas(3) but the cluster has only 2 brokers?
    Topic creation fails — the requested replication factor exceeds available brokers. AdminClient returns an error; by default the app still starts (creation error is logged, non-fatal) unless fail-fast is enabled.
  • You changed RETENTION_MS in a NewTopic bean, redeployed, but the topic's retention didn't change. Why?
    TopicBuilder configs are applied only at topic creation. For an existing topic KafkaAdmin ignores config differences unless setModifyTopicConfigs(true) (spring.kafka.admin.modify-topic-configs) is enabled.

saying these in an interview costs you the question

  • Thinking config values can be numeric types rather than Strings
  • Assuming omitted partitions/replicas inherit broker defaults (they default to 1/1)
  • Believing config changes on an existing topic apply automatically on redeploy

context

resolving links…