skip to content

What is Kafka Connect, and what problem does it solve compared to writing your own producer/consumer applications?

level: juniorimportance: must knowfreq 70%

answer

  1. framework, not custom code
  2. source = into Kafka; sink = out of Kafka
  3. connector → tasks → workers
  4. converters handle format
  5. not a processing engine

basics

~10 s

Kafka Connect is a framework for streaming data between Kafka and external systems (databases, files, S3) using ready-made connectors, so you configure plugins instead of writing custom producer/consumer code.

solid answer

~40 s

Kafka Connect is a runtime and framework, shipped with Apache Kafka, for moving data between Kafka and external systems without writing bespoke code. Source connectors pull data from systems like databases into Kafka topics; sink connectors push from topics into systems like Elasticsearch or S3. You deploy reusable connector plugins and configure them with JSON/properties — declaring topics, converters, and connection details. Connect handles the hard cross-cutting concerns for you: scalability via worker clusters, fault tolerance and rebalancing, offset tracking so work resumes after a restart, at-least-once (or exactly-once for sources) delivery, schema/format conversion via converters, and a REST API for management. Writing your own producers/consumers means re-implementing all of that per integration; Connect standardizes it.

go deeper

for a junior

Know the one-line definition and the source-vs-sink direction.

for a middle

Explain connector/task/worker/converter roles and why Connect beats hand-rolled producers.

for a senior

Discuss delivery guarantees, SMT limits vs. Streams, and when Connect is the wrong tool.

for a principal

Frame Connect's place in a data platform: standardized ingest/egress layer, schema governance via converters/registry, and ops model across teams.

## What Kafka Connect is **Apache Kafka** is a distributed log/messaging system. **Kafka Connect** is a separate component bundled with Kafka whose only job is to move data *between Kafka and other systems* reliably and at scale, without you writing integration code. ### The two directions - **Source connector**: reads from an external system (e.g., a PostgreSQL table, a file, a CDC stream) and *produces* records into Kafka topics. - **Sink connector**: *consumes* from Kafka topics and writes to an external system (e.g., S3, Elasticsearch, JDBC). ### Key vocabulary - **Connector**: a logical job, configured by you, that defines what to copy and how. A connector instance splits work into **tasks**. - **Task**: the unit of parallelism that actually moves data. A connector with `tasks.max=4` may run up to 4 tasks across the cluster. - **Worker**: a JVM process running the Connect runtime; it hosts and executes tasks. - **Converter**: serializes/deserializes record keys and values to/from a wire format (JSON, Avro, Protobuf) — set via `key.converter`/`value.converter`. - **Plugin**: the packaged connector/converter/transform JARs the worker loads from `plugin.path`. ### Why not just write a producer/consumer? A hand-rolled integration forces you to re-solve: parallelism and load balancing, restart/resume (offset bookkeeping), failure handling and retries, scaling out, schema conversion, and operational control (start/stop/reconfigure). Connect provides all of these as a **framework**: you write configuration, not plumbing. For common systems a maintained connector already exists, so often you write *zero* code. ### Edge cases / nuances - Connect is **not** a stream-processing engine — for joins/aggregations/transformations beyond simple per-record **Single Message Transforms (SMTs)**, use Kafka Streams or ksqlDB. - Source connectors can achieve **exactly-once** (KIP-618, `exactly.once.source.support`); sinks are typically **at-least-once** and need idempotent writes downstream.

  • What is the difference between a source connector and a sink connector?
    A source connector imports data from an external system into Kafka topics (it produces); a sink connector exports data from Kafka topics into an external system (it consumes).
  • When would you NOT use Kafka Connect?
    When you need real stream processing — joins, windowed aggregations, enrichment across streams. Connect only does ingest/egress plus light per-record SMTs; use Kafka Streams or ksqlDB for processing.

saying these in an interview costs you the question

  • Calling Connect a stream-processing engine like Kafka Streams.
  • Saying you must write Java code for every integration (maintained connectors are configured, not coded).
  • Confusing source/sink direction (source = into Kafka).

context

resolving links…