What is Kafka Connect, and how do sink connectors get data from Kafka topics into a data lake or warehouse like S3, BigQuery, or Snowflake?
answer
- framework not custom code
- source in / sink out
- tasks.max = parallelism
- converter + Schema Registry
- at-least-once by default
basics
~20 sKafka Connect is a framework for moving data between Kafka and external systems without custom code. A sink connector reads records from Kafka topics and writes them to a destination like S3, BigQuery, or Snowflake using ready-made plugins and config.
solid answer
~40 sKafka Connect is a runtime and framework for streaming data into and out of Kafka declaratively via JSON config instead of bespoke producer/consumer code. A source connector pulls data into Kafka; a sink connector reads from topics and writes to an external system. For lakehouse ingestion you run sink connectors such as the Confluent S3 sink, BigQuery sink, or Snowflake sink. Connect runs in distributed mode as a cluster of workers; each connector is split into tasks for parallelism, and offsets are committed so delivery is at-least-once by default. Sinks buffer records and flush in batches (e.g., S3 sink writes files partitioned by topic/partition/time). Converters (Avro/JSON/Protobuf, often with Schema Registry) handle serialization, and Single Message Transforms (SMTs) can reshape records in flight.
go deeper
Know Connect = no-code framework, sink = topic to external system, and name S3/BigQuery/Snowflake sinks.
Explain workers, tasks.max parallelism, converters + Schema Registry, and the at-least-once flush/commit cycle.
Discuss partitioners, file formats, DLQ, SMTs, and how at-least-once duplicates are handled downstream.
Reason about Connect cluster sizing, schema-evolution governance, and when a managed sink beats a bespoke streaming job.
## What problem Kafka Connect solves Writing a one-off consumer to copy every Kafka topic into a warehouse is repetitive, error-prone work: you re-solve batching, retries, offset tracking, schema handling, and scaling each time. **Kafka Connect** is a framework (shipped with Apache Kafka) that turns this into configuration. You declare *what* to connect and Connect handles *how*. ## Source vs sink - A **source connector** brings external data *into* Kafka topics (e.g., a database into Kafka). - A **sink connector** reads *from* Kafka topics and writes data *out* to an external system (S3, BigQuery, Snowflake, Iceberg, Elasticsearch, JDBC databases). For lakehouse/warehouse ingestion you almost always use **sink connectors**. ## How a sink works mechanically 1. **Workers**: Connect runs as a cluster of JVM processes called *workers*. In **distributed mode** they coordinate via internal Kafka topics that store connector configs, offsets, and status, so the cluster is fault-tolerant and rebalances if a worker dies. 2. **Connector → tasks**: A connector is configured once but is split into N **tasks** (`tasks.max`) for parallelism. Connect assigns topic partitions across tasks much like a consumer group, so throughput scales with partitions. 3. **Converters**: Bytes on the wire must be deserialized. A **converter** (e.g., `AvroConverter`, `JsonConverter`, `ProtobufConverter`) turns raw bytes into Connect's internal record format. With a **Schema Registry**, Avro/Protobuf converters validate and evolve schemas. 4. **SMTs (Single Message Transforms)**: Lightweight per-record transforms (rename fields, add a timestamp, route topics, mask data) applied before the record reaches the sink. 5. **Buffering and flush**: Sinks accumulate records and **flush in batches**. The S3 sink, for example, groups records by topic/partition and a partitioner (default by Kafka partition, or `TimeBasedPartitioner` for date paths like `year=/month=/day=/hour=`) and writes objects when `flush.size` records accumulate or a `rotate.interval.ms` elapses, in formats like Parquet, Avro, or JSON. 6. **Offset commit**: After a successful flush, Connect commits consumer offsets. Because the data is written *then* offsets committed, the default guarantee is **at-least-once**: a crash between write and commit can replay records, producing duplicates downstream. ## Why this matters for analytics Sink connectors give you a managed, scalable, schema-aware bridge from the streaming layer to the analytics layer with retries, dead-letter queues (`errors.deadletterqueue.topic.name`), and monitoring built in — no custom ingestion service to maintain.
- What delivery guarantee does a Connect sink give by default, and why?At-least-once: the sink writes the batch to the destination, then commits offsets. A failure between those steps replays records, so downstream can see duplicates unless the sink supports exactly-once (e.g., via idempotent upserts or transactional writes).
- What is the role of a converter versus an SMT?A converter (de)serializes the whole record key/value to/from bytes, usually tied to a wire format and Schema Registry. An SMT is a small per-record transformation (rename, mask, route, add field) applied after deserialization, before the connector hands the record to the sink.
saying these in an interview costs you the question
- Saying Connect requires writing custom producer/consumer code — its whole point is config-driven, no code.
- Claiming sinks are exactly-once by default — they are at-least-once unless the connector adds dedup/transactional support.
- Confusing source (into Kafka) and sink (out of Kafka) direction.