skip to content

Should you rely on Spring's KafkaAdmin auto-creation for provisioning topics in production? Discuss the trade-offs and how you'd structure it.

level: principalimportance: nice to knowfreq 25%

answer

  1. additive reconciler, not desired-state
  2. no RF change, no delete, no shrink
  3. modifyTopicConfigs = silent prod mutation
  4. 1/1 default footgun
  5. infra-as-code for prod, beans for dev

basics

~20 s

It's great for local/dev and convenience, but as the sole production strategy it's weak: it won't change replication factor, tolerates config drift by default, can silently create 1/1 topics, and ties correct topic layout to app startup. Prefer infra-as-code for prod and use KafkaAdmin declaratively.

solid answer

~40 s

KafkaAdmin auto-creation is declarative and convenient, but it's an *additive reconciler*, not a desired-state controller. Limitations for production: it never changes replication factor or deletes topics, config drift is ignored unless modifyTopicConfigs is on (and then a redeploy can silently mutate prod retention), forgotten partitions/replicas default to a fragile 1/1, and topic correctness becomes coupled to app startup and broker availability. Many orgs disable app-side auto-creation in prod (setAutoCreate(false) or don't declare NewTopic beans there) and provision topics via infra-as-code — Terraform Kafka provider, GitOps, kafka-topics.sh in a pipeline — where RF, min.insync.replicas, and partitions are reviewed and versioned centrally. Where you do use NewTopic beans, keep them environment-aware (profiles/@ConditionalOnProperty), pin replicas and min.insync.replicas explicitly, and treat them as one input alongside the platform's topic policy, not the source of truth.

code

java · 22 lines
java
// Keep declarative topics for non-prod; let infra-as-code own prod topics.
@Configuration
class TopicConfig {

    @Bean
    @ConditionalOnProperty(name = "app.kafka.declare-topics", havingValue = "true")
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
                .partitions(12)
                .replicas(3)                                    // explicit, never rely on 1/1
                .config(TopicConfig.MIN_INSYNC_REPLICAS_CONFIG, "2")
                .build();
    }

    // In prod, disable app-side provisioning entirely so beans (if any)
    // only document intent and Kafka platform tooling is the source of truth.
    @Bean
    @Profile("prod")
    KafkaAdmin.KafkaAdminCustomizer noAppSideProvisioning() {
        return admin -> admin.setAutoCreate(false);
    }
}

go deeper

for a junior

Know KafkaAdmin auto-creation is mainly a dev/local convenience.

for a middle

List a couple of prod limitations (1/1 default, config drift) and prefer explicit RF.

for a senior

Articulate the additive-reconciler gaps and gate beans by profile/property.

for a principal

Define layered dev-vs-prod strategy and central topic governance, treating RF/reassignment as platform operations.

## Why teams reach for KafkaAdmin auto-creation - **Zero-friction dev/test:** declare a NewTopic @Bean and the topic exists on `bootRun` or against an embedded/Testcontainers broker. - **Co-located intent:** the topic's shape lives next to the code that uses it. - **Idempotent-ish:** re-running against an existing topic is safe (it won't recreate). ## Why it's insufficient as the *sole* production strategy 1. **Not a full desired-state controller.** It only does additive reconciliation: increase partitions, optionally alter declared configs. It **never** changes replication factor, never deletes, never shrinks. Your beans can't express the real desired end state. 2. **Silent config mutation risk.** With `modifyTopicConfigs=true`, a redeploy can quietly change production `retention.ms`/`cleanup.policy` because someone edited a bean — a data-loss-shaped change with no review gate. With it off, you get **config drift** instead. 3. **The 1/1 footgun.** Omitting partitions/replicas yields a single-partition, single-replica topic — no parallelism, no durability — and it's easy to miss in review. 4. **Startup coupling.** Correct topic layout now depends on the broker being reachable at the right moment and on your service winning the race against any producer that might lazily create a mis-shaped topic (if broker auto-create is on). No retry means a transient outage can leave topics uncreated until restart. 5. **Ownership diffusion.** If every service declares its own topics, there's no central, reviewable policy for RF, `min.insync.replicas`, partition counts, or naming — critical for durability/compliance. ## A pragmatic structure - **Local/dev/test:** use NewTopic beans / KafkaAdmin.NewTopics freely; embedded broker + Testcontainers make this fast. - **Production:** provision topics via **infra-as-code** — Terraform Kafka provider, a GitOps topic manifest, or a pipeline step running `kafka-topics.sh` / AdminClient — where partitions, RF, `min.insync.replicas`, retention, and naming conventions are versioned and peer-reviewed centrally. (This is cluster/platform operations, a sibling concern to the app.) - **If you keep NewTopic beans in prod:** make them environment-aware and safe: ```java @Bean @Profile("!prod") // or @ConditionalOnProperty("app.kafka.declare-topics") NewTopic ordersTopic() { return TopicBuilder.name("orders").partitions(12).replicas(3) .config(TopicConfig.MIN_INSYNC_REPLICAS_CONFIG, "2").build(); } ``` or globally `setAutoCreate(false)` in prod so beans document intent without provisioning. ## What a strong answer signals - Distinguishes **declarative convenience** from **desired-state infrastructure management**. - Knows the concrete gaps (RF, deletes, drift, 1/1, startup coupling). - Proposes a layered approach (dev vs prod) and central topic policy, and recognizes RF/reassignment is broker/cluster operations, not app responsibility.

  • If topics are managed by Terraform/GitOps in prod, why keep NewTopic beans at all?
    For fast local/dev/test loops (bootRun, embedded/Testcontainers brokers) and as living documentation of a service's topic contract. You gate them off in prod (profiles/@ConditionalOnProperty or setAutoCreate(false)) so the platform tooling stays the single source of truth.
  • What's the danger of enabling modifyTopicConfigs=true in production?
    A code change to a NewTopic bean's config (e.g., retention.ms) will be applied to the live topic on the next deploy with no infra review gate — potentially shrinking retention and causing premature data deletion. Config changes bypass the review rigor infra-as-code would enforce.

saying these in an interview costs you the question

  • Treating KafkaAdmin as a full desired-state topic manager for prod
  • Enabling modifyTopicConfigs in prod without appreciating the silent-mutation risk
  • Assuming replication factor / deletions are handled by beans
  • Letting each service own topic policy with no central review

context

resolving links…