Ecosystem and Platform Choices

librdkafka is a C/C++ implementation of the Kafka client protocol. The official Python, Go, .NET, and Node.js clients are thin language bindings that wrap librdkafka, so they share one battle-tested core instead of each reimplementing Kafka.

Kafka Connect is a framework for moving data between Kafka and external systems without custom code. A sink connector reads records from Kafka topics and writes them to a destination like S3, BigQuery, or Snowflake using ready-made plugins and config.

Azure Event Hubs exposes a Kafka-compatible endpoint on port 9093, so a normal Kafka client can produce and consume by just changing bootstrap.servers and using SASL/SSL auth with a connection string — no Kafka broker is actually run.

They are alternative streaming systems that speak the Kafka wire protocol, so existing Kafka clients work unchanged. Redpanda is a C++ broker with no JVM or ZooKeeper; WarpStream stores data in S3 with stateless brokers.

Kafka stores messages as an append-only log that consumers read by position (offset); messages stay for a retention period. A traditional queue deletes each message once a consumer acknowledges it, so reading is destructive.

A STREAM is an unbounded, append-only sequence of independent events (an immutable log). A TABLE represents the latest value per key (a mutable, upserted view). Use a STREAM for facts/events, a TABLE for current state.

A managed Kafka offering is Kafka run for you by a cloud vendor (e.g. Confluent Cloud, Amazon MSK, Aiven). The vendor handles installing brokers, patching, scaling, backups, and hardware failures, so you mostly just create topics and connect clients.

Kafka Streams is a Java library you embed inside your own application, with no separate cluster. Flink and Spark are standalone clusters you deploy, scale, and operate on the side, each with its own master and worker processes.

Pulsar offers four subscription modes — Exclusive, Failover, Shared, and Key_Shared. Shared/Key_Shared fan messages out across consumers like a work queue; Exclusive/Failover preserve per-partition order like a streaming log. Kafka only has the streaming-log model via consumer groups.

In Kafka each consumer group has its own offsets, so many groups read the same topic without copying data. RabbitMQ/SNS fan out by duplicating each message into multiple queues at publish time.

CSAS and CTAS create a new, continuously-updated derived stream or table from a SELECT over existing ones. Each launches a persistent query — a long-running Kafka Streams job — that writes results into a new backing Kafka topic.

A push query (EMIT CHANGES) subscribes to a continuous stream of updates and never finishes until cancelled — it pushes new results as they arrive. A pull query is a point-in-time lookup against a materialized table's current state and returns immediately.

A schema-aware serializer registers the schema in Schema Registry, gets back an integer schema ID, and writes a small header — a 0x00 magic byte plus the 4-byte big-endian schema ID — in front of the serialized payload. Any client (JVM or librdkafka) that follows this same wire format can look up the ID and deserialize, which is what makes Avro/Protobuf data interoperable across languages.

Debezium is a Kafka Connect source connector that reads a database's transaction log and emits each insert/update/delete as a Kafka event. Sink connectors then land those change events in the lake, giving near-real-time, low-load, complete change history instead of nightly full-table dumps.

Protocol-compatible endpoints implement the produce/consume path but often lack full Kafka transactions/exactly-once, log compaction (cleanup.policy=compact), and parts of the AdminClient. Detect gaps by testing transactional.id, compacted topics, and admin calls against the endpoint before migrating.

Kafka Streams uses Kafka transactions (set processing.guarantee=exactly_once_v2) so reads, state updates, and writes commit atomically. Flink uses checkpoint barriers plus a two-phase-commit Kafka sink. Spark uses checkpointed offsets with idempotent/transactional sinks.

Pick Kafka Streams for Kafka-to-Kafka microservice logic owned by one team with no extra cluster. Pick Flink for large, complex, low-latency stateful jobs and multi-source joins on a shared platform. Pick Spark when you already run Spark and want unified batch plus streaming SQL.

The Confluent REST Proxy is an HTTP service that sits in front of a Kafka cluster, letting clients produce and consume via REST/JSON instead of the binary Kafka protocol. Use it for environments that can't run a native client — restricted languages, serverless/edge, or firewalled HTTP-only networks — at the cost of higher latency and lower throughput.

Kafka acts as the central event highway: producers and CDC feed raw events in, sink connectors land them in cheap lake storage, and transformations run afterward in the warehouse/lake (the 'T' of ELT). Table formats like Apache Iceberg make those landed files behave like real, queryable, transactional tables.

The S3 sink groups records by partitioner (by Kafka partition or by time) and writes batched objects (Parquet/Avro/JSON) when flush.size or a rotate interval is hit. It achieves exactly-once by naming files deterministically from offsets, so replays overwrite the same object instead of duplicating.

An AMQP or MQTT edge is a gateway that accepts those protocols at the boundary and forwards messages into Kafka. Use them when clients can't speak Kafka — e.g. IoT/MQTT devices or AMQP messaging apps — trading native Kafka features for broad client reach.

Use a Pub/Sub-to-Kafka bridge to connect Google Pub/Sub and Kafka ecosystems without rewriting apps — e.g. fan messages from Pub/Sub into Kafka topics. Trade-offs: extra hop adds latency, the bridge can break ordering/exactly-once, and it's another component to run.

On Apache Kafka you size brokers, disks, and partitions. On Event Hubs you buy Throughput Units (or Processing/Capacity Units) that cap MB/s and events/s regardless of partition count; partitions are fixed at creation and mainly affect parallelism, not capacity.

Redpanda is one C++ binary using a thread-per-core model, with no JVM (no GC pauses) and no ZooKeeper. It uses Raft for both metadata and partition replication, giving lower, more predictable tail latency and simpler operations than JVM-based Kafka.

A Kafka consumer tracks a position (offset) and can seek backward to an earlier offset or timestamp to re-read records that are still within retention. A queue deletes messages on ack, so there's nothing left to re-read.

Provisioned MSK gives you sized broker instances you pick and pay for by the hour; you control instance type and broker count. MSK Serverless hides brokers and capacity entirely and bills on throughput/storage, auto-scaling. Choose Serverless for spiky/unknown load, provisioned for steady high throughput where per-unit cost is lower.

Kafka Streams keeps state in embedded RocksDB on each instance, backed up to compacted changelog topics in Kafka. Flink keeps state in a state backend (heap or RocksDB) snapshotted to durable storage via checkpoints. Spark stores state in a checkpointed, versioned state store.

Pulsar bakes in a tenant/namespace/topic hierarchy with per-tenant quotas, auth, and isolation, and supports cross-region replication by configuring clusters on a namespace. Kafka has a flat topic namespace and relies on external tools (ACLs, MirrorMaker 2) to approximate both.

In Kafka a partition's whole log lives on the broker that leads it, so the partition is the storage unit. In Pulsar a topic is a sequence of small segments (BookKeeper ledgers) spread across many bookies, so no single node holds the whole topic.