skip to content

What do auto.create.topics.enable and num.partitions control, and why is auto-creation often disabled in production?

level: middleimportance: should knowfreq 60%

answer

  1. broker creates on first reference
  2. defaults: 1 partition / RF 1
  3. typos -> junk topics
  4. no fault tolerance with RF=1
  5. allow.auto.create.topics = client side

basics

~20 s

auto.create.topics.enable lets the broker auto-create a missing topic when a client produces to or fetches from it. num.partitions and default.replication.factor decide the new topic's shape. It's disabled in prod to prevent typo'd or unplanned topics.

solid answer

~40 s

auto.create.topics.enable is a broker-level config (default true in many distros) that creates a topic on demand when a producer sends to, a consumer subscribes to, or metadata is requested for a non-existent topic. The auto-created topic uses the broker defaults num.partitions (default 1) and default.replication.factor (default 1). Production teams disable it because: a typo in a topic name silently creates a junk topic; auto-created topics get the default 1 partition / RF 1, which has no fault tolerance and poor parallelism; and it bypasses governance/naming policies. Instead, topics are provisioned explicitly via AdminClient or infrastructure-as-code. Note: even with it enabled, internal topics and consumer subscriptions can trigger creation, so the safe production stance is auto.create.topics.enable=false plus explicit creation.

go deeper

for a junior

Know auto-create makes topics on demand and defaults to 1 partition / RF 1.

for a middle

Explain why prod disables it and the difference between broker and client auto-create settings.

for a senior

Tie it to governance, IaC topic provisioning, and the failure mode of silent typo topics.

for a principal

Define org-wide topic provisioning policy: auto-create off, declarative management, naming/ACL standards, capacity defaults.

## auto.create.topics.enable This is a **broker-side** configuration. When `true`, the broker will transparently create a topic the first time a client references a name that doesn't exist — specifically when a producer sends a record to it, a consumer subscribes/fetches metadata for it, or a metadata request names it. The created topic inherits the broker's default shape. ## num.partitions and default.replication.factor These are the **broker defaults** used whenever a topic is created without an explicit shape — whether by auto-creation or by a `--create` that omits `--partitions`/`--replication-factor`. - `num.partitions` default is **1**. - `default.replication.factor` default is **1**. So an auto-created topic typically has 1 partition and RF=1 — meaning **no replication, no fault tolerance, and no parallelism**. If that broker's disk fails, the data is gone. ## Why disable in production 1. **Typos become topics.** Producing to `order-events` when the real topic is `orders-events` silently creates a useless topic that quietly swallows data. 2. **Bad defaults.** RF=1 / 1 partition is almost never what you want; the topic can't survive a broker loss and can't scale consumers. 3. **Governance.** Naming conventions, retention policies, ACLs, and capacity planning are bypassed. 4. **Surprise load.** A misbehaving client can spawn many topics. ## The production pattern Set `auto.create.topics.enable=false` on brokers and provision topics explicitly: ```java NewTopic t = new NewTopic("orders", 6, (short) 3); admin.createTopics(List.of(t)); ``` or via Terraform/declarative topic management. With auto-create off, producing to a missing topic returns UNKNOWN_TOPIC_OR_PARTITION, surfacing the mistake immediately. ## Edge cases - Some managed clouds (e.g. certain Confluent Cloud tiers) force auto-create off and ignore the setting. - Consumer subscription to a non-existent topic can also trigger auto-create; `allow.auto.create.topics` is a *client-side* consumer config (default true) that lets consumers suppress this independently.

  • What error does a producer get when auto-create is off and the topic doesn't exist?
    It fails with UNKNOWN_TOPIC_OR_PARTITION, which makes the missing/mistyped topic immediately visible rather than silently created.
  • Is there a way for a consumer to avoid triggering auto-creation even when the broker allows it?
    Yes — set the client-side consumer config allow.auto.create.topics=false so subscriptions don't request broker-side creation.

saying these in an interview costs you the question

  • Saying auto-created topics get RF=3 or 'good' defaults (they default to RF=1, 1 partition)
  • Confusing broker-side auto.create.topics.enable with client-side allow.auto.create.topics
  • Claiming auto-create is safe because it 'just works'
  • Thinking num.partitions affects existing topics (it only applies to newly created ones)

context

resolving links…