Azure Event Hubs exposes a 'Kafka protocol surface.' What does that mean, and what Kafka features or behaviors should you NOT assume are present?
answer
- Kafka wire protocol over Azure engine, not real brokers
- namespace≈cluster, event hub≈topic
- SASL_SSL + connection string at :9093
- Don't assume EOS/transactions, compaction, full AdminClient
- Partitions often fixed at creation; tiers gate features
basics
~20 sEvent Hubs isn't Apache Kafka; it's Azure's own event service that speaks the Kafka wire protocol. Existing Kafka clients can produce/consume by changing the bootstrap endpoint. But it's not a full Kafka broker, so some features (Kafka Connect-style admin, transactions, compacted topics, certain configs) may be limited or absent.
solid answer
~50 sAzure **Event Hubs** is a native Azure streaming service that offers a **Kafka-compatible endpoint**: it implements the Kafka producer/consumer wire protocol (roughly Kafka 1.0+ APIs) so unmodified Kafka clients connect by pointing the bootstrap server at the Event Hubs namespace and using SASL with a connection string. An Event Hubs **namespace** maps to a Kafka cluster, an **event hub** to a topic, and **partitions** map to partitions. Because it is a re-implementation, not Kafka brokers, you should not assume full parity: historically **log compaction**, **idempotent/transactional producers** (exactly-once), some **AdminClient** operations, and certain broker configs were unsupported or behaved differently; partition counts are often fixed at creation; and there is no real ZooKeeper/KRaft you control. The Kafka ecosystem (Kafka Streams, Connect connectors) may work for basic cases but isn't guaranteed. The value is reusing Kafka clients while staying inside Azure's billing, IAM (Entra ID), and tiers (Standard/Premium/Dedicated). Validate feature support against your version rather than assuming Apache semantics.
go deeper
Know Event Hubs lets Kafka clients connect by changing the endpoint, but it isn't full Kafka.
Map namespace/event-hub/partition to Kafka concepts and recall the SASL_SSL connection method.
Enumerate non-parity features (EOS, compaction, AdminClient, partition changes) and verify against tier before adopting.
Govern migration risk: inventory app-level Kafka guarantees, weigh Azure-native lock-in/identity benefits vs feature gaps.
## What 'protocol surface' means Apache Kafka defines a **wire protocol**: a set of binary request/response APIs (Produce, Fetch, Metadata, OffsetCommit, etc.) that clients speak over TCP. Azure **Event Hubs** is a separate, Azure-native streaming product, but Microsoft implemented an **endpoint that speaks this Kafka protocol**. So a standard Kafka client library (Java, librdkafka, etc.) can talk to Event Hubs by: - pointing `bootstrap.servers` at `<namespace>.servicebus.windows.net:9093`, - using **SASL_SSL** with mechanism `PLAIN` and a connection string as the password (or Entra ID/OAuth). No code rewrite, just config. Conceptually: **namespace ≈ Kafka cluster**, **event hub ≈ topic**, **partitions ≈ partitions**, **consumer groups ≈ consumer groups**. ## Why it is NOT full Kafka It is a **compatibility layer over a different engine**, not Apache Kafka brokers. Consequences to watch: - **Log compaction**: compacted topics (key-based retention) were historically unsupported; support has been added in some tiers but should be verified, not assumed. - **Transactions / idempotence**: Kafka's exactly-once (idempotent producer, transactional API) is not guaranteed; many setups support at-least-once only. - **AdminClient / configs**: some administrative operations and per-topic broker configs behave differently or are not honored. Partition count is typically **fixed at creation** and may not be increasable the Kafka way. - **No ZooKeeper/KRaft** you can touch — metadata is Azure-internal. - **Ecosystem tools** (Kafka Streams, Connect, ksqlDB) may run for simple cases but are not officially full-coverage; advanced features that rely on transactions or compaction can break. - **Tiers** matter: **Standard**, **Premium**, and **Dedicated** differ in throughput units / processing units, retention, and which features (e.g. compaction, larger messages) are available. ## Operational model Event Hubs abstracts brokers, scaling (throughput units or auto-inflate, processing units on Premium/Dedicated), patching, and HA. Auth/identity is Azure-native (**Entra ID / SAS connection strings**), and billing is by tier plus throughput/processing units and capture/retention — not by broker-hour. ## How to reason about it Treat Event Hubs' Kafka surface as **'Kafka clients work, Kafka guarantees may not.'** Before adopting it for an existing Kafka app, enumerate the features your app relies on (EOS, compaction, AdminClient calls, dynamic partition increase, specific configs) and verify each against the target tier and current docs. It shines when you want Azure-native operations/billing/identity and only need core produce/consume semantics.
- An existing Kafka app uses the transactional producer for exactly-once. Is Event Hubs' Kafka surface a safe drop-in?Not safely. Kafka transactions/idempotent EOS are not guaranteed on the Event Hubs Kafka surface. You must verify support for your tier/version; many configurations provide at-least-once only, so the app's exactly-once assumption may break.
- How does a Kafka client actually connect to Event Hubs?Point bootstrap.servers at <namespace>.servicebus.windows.net:9093, use security.protocol=SASL_SSL with SASL mechanism PLAIN, username '$ConnectionString', and the namespace connection string as the password (or use Entra ID/OAuth).
saying these in an interview costs you the question
- Calling Event Hubs 'managed Apache Kafka' — it is a Kafka-protocol-compatible surface over a different engine.
- Assuming exactly-once (transactions/idempotence) and log compaction work identically.
- Assuming you can increase partitions or set arbitrary broker configs like on real Kafka.
- Believing the full Kafka ecosystem (Streams/Connect/ksqlDB) is guaranteed to work.