skip to content

Compare standalone mode and distributed mode in Kafka Connect. When would you choose each, and how does state/configuration storage differ?

level: middleimportance: must knowfreq 75%

answer

  1. standalone = 1 worker, file offsets
  2. distributed = group.id cluster
  3. 3 internal topics: config/offset/status
  4. rebalance on worker death
  5. REST API in distributed

basics

~20 s

Standalone runs a single worker and stores offsets in a local file — simple, no fault tolerance, good for dev. Distributed runs multiple coordinated workers that store config, offsets, and status in Kafka topics, giving scalability and fault tolerance.

solid answer

~40 s

Connect has two execution modes. **Standalone** runs one worker process; connector configs are passed as properties files on startup, and offsets are persisted to a local file via `offset.storage.file.filename`. It has no fault tolerance or scale-out and is meant for development, testing, or a single-agent collector. **Distributed** runs one or more workers that form a cluster via a shared `group.id` (they join a Kafka consumer-group-style coordination protocol). All state lives in three internal compacted Kafka topics: `config.storage.topic` (connector/task configs), `offset.storage.topic` (source offsets), and `status.storage.topic` (connector/task state). Connectors are managed through the REST API, not files. If a worker dies, the cluster rebalances its tasks onto survivors. Choose distributed for any production deployment; standalone only for local or single-node edge cases.

go deeper

for a junior

Know standalone = single/dev with file offsets, distributed = cluster/prod with topics.

for a middle

Name the three internal topics and how group.id forms the cluster.

for a senior

Discuss compaction, replication, partition counts, and rebalance behavior on failure.

for a principal

Reason about HA topology, single-node distributed pattern, and multi-cluster isolation via group.id/topic naming.

## Two ways to run the Connect runtime A **worker** is the JVM process running Connect. How workers coordinate (or don't) defines the mode. ### Standalone mode - A **single worker process**, started with `connect-standalone.sh worker.properties connector1.properties connector2.properties ...`. - Connector configs are supplied as **properties files at launch**. - **Offsets** (how far a source connector has read) are stored in a **local file**: `offset.storage.file.filename`. Status and config have no durable cluster store — they live only in that process. - **No fault tolerance, no scale-out**: if the process dies, work stops; restarting resumes from the local offset file. - Good for: development, quick tests, demos, or a genuinely single-node collector (e.g., one agent tailing a file on one host). ### Distributed mode - One or more **workers** started with `connect-distributed.sh worker.properties`. - Workers with the **same `group.id`** form a **cluster**. They use Kafka's group-membership/rebalance protocol (the same machinery consumer groups use) to elect a leader and assign connectors and tasks across workers. - **State lives in three internal Kafka topics** (all should be **compacted**): - `config.storage.topic` — connector and task configurations. Must have **replication factor ≥ 3** in prod and is created with a **single partition** (config must be totally ordered). - `offset.storage.topic` — source-connector offsets. Multiple partitions (e.g., 25), high replication. - `status.storage.topic` — connector/task/worker status (RUNNING, FAILED, PAUSED). Multiple partitions, high replication. - Connectors are managed via the **REST API** (`POST /connectors`, `PUT /connectors/{name}/config`, etc.), and config propagates to all workers through the config topic. - **Fault tolerance**: a dead worker triggers a **rebalance**; its tasks move to surviving workers. Scale-out: add a worker with the same `group.id` and it joins the cluster. ### How to choose | Need | Mode | |---|---| | Production, HA, scale | Distributed | | Dev/test, single host | Standalone | | Centralized REST management | Distributed | | Zero Kafka-side topics, file offsets | Standalone | ### Edge cases / gotchas - A 'single-node distributed' cluster (one worker, distributed mode) is a common production pattern when you want REST management + topic-backed state but only need one box; it still gets durable config/offset/status in Kafka. - Internal topics **must be compacted** — if auto-created with the wrong cleanup policy, config history can be lost. Pre-create them in locked-down clusters. - Two clusters must use **different `group.id`** *and* different internal topic names; sharing them corrupts state.

  • In distributed mode, where are connector configurations stored, and why must that topic be compacted and single-partition?
    In the `config.storage.topic`. Compaction keeps the latest config per key durable; a single partition guarantees total ordering of config updates so all workers converge on the same state.
  • Can you run distributed mode with only one worker? Why might you?
    Yes. You still get topic-backed config/offset/status durability and REST management, which standalone lacks — useful when you want production-grade state handling but only one node currently.

saying these in an interview costs you the question

  • Saying standalone stores offsets in Kafka topics (it uses a local file).
  • Claiming distributed mode needs ZooKeeper for coordination (it uses Kafka's group protocol/topics).
  • Saying you pass connector configs as files in distributed mode (you use the REST API).

context

resolving links…