skip to content

When would you put a Google Pub/Sub-to-Kafka bridge (or the Pub/Sub Kafka shim) between a publisher and a Kafka cluster, and what are the trade-offs?

level: middleimportance: should knowfreq 30%

answer

  1. bridge = ferry Pub/Sub <-> Kafka
  2. Pub/Sub Group Kafka Connector (source/sink)
  3. ordering keys vs partition order mismatch
  4. at-least-once, EOS doesn't survive
  5. use for integration/migration, not hot path

basics

~20 s

Use a Pub/Sub-to-Kafka bridge to connect Google Pub/Sub and Kafka ecosystems without rewriting apps — e.g. fan messages from Pub/Sub into Kafka topics. Trade-offs: extra hop adds latency, the bridge can break ordering/exactly-once, and it's another component to run.

solid answer

~50 s

A Pub/Sub↔Kafka bridge moves messages between Google Cloud Pub/Sub and Apache Kafka so teams can keep their existing producers/consumers across both ecosystems. Patterns: the **Pub/Sub Group Kafka Connector** (Kafka Connect source/sink connectors that publish to / read from Pub/Sub), and **Pub/Sub's Kafka-protocol compatibility** layer that lets Kafka clients talk to Pub/Sub-style backends. You'd use it to ingest cloud-native Pub/Sub events into an on-prem/Kafka analytics pipeline, or to let a Kafka-based app emit into GCP services. Trade-offs: an extra network hop and component to operate and monitor; ordering guarantees differ (Pub/Sub ordering keys vs Kafka partition order) and aren't 1:1; exactly-once and transactions usually don't survive the bridge (at-least-once with duplicates is typical); offset/ack-deadline semantics differ; and you inherit two systems' quotas and failure modes. Prefer it for integration/migration, not as a permanent hot path if latency and EOS matter.

go deeper

for a junior

Know a bridge connects Pub/Sub and Kafka so apps interoperate.

for a middle

Name the connector pattern (source/sink) and the latency/ordering/at-least-once trade-offs.

for a senior

Reason about ack-deadline vs offset semantics, backpressure/redelivery, and when to avoid the bridge.

for a principal

Decide bridge-vs-native per workload, balancing migration value against latency, EOS, and lock-in.

## The two ecosystems **Apache Kafka**: partitioned log; ordering is per-partition; consumers track **offsets**; supports transactions/EOS and compaction. **Google Cloud Pub/Sub**: a fully managed pub/sub message bus with **topics** and **subscriptions**; consumers **ack** individual messages within an **ack deadline**; ordering is optional via **ordering keys**; scaling is automatic and serverless. They have different delivery and ordering models. ## What a bridge is A **bridge** is a component that ferries messages between the two so applications written for one can interoperate with the other: - **Pub/Sub Group Kafka Connector**: open-source **Kafka Connect** connectors — a *source* connector reads from Pub/Sub and writes to Kafka topics, and a *sink* connector reads from Kafka and publishes to Pub/Sub. Runs on Kafka Connect workers. - **Pub/Sub Kafka-protocol shim / compatibility**: lets standard Kafka clients speak to a Pub/Sub-backed endpoint, analogous to Event Hubs' approach, so you don't rewrite to the Pub/Sub SDK. ## When to use it 1. **Hybrid/migration**: events originate in GCP Pub/Sub but your analytics, stream processing, or storage lives in Kafka (or vice-versa). The bridge avoids rewriting either side. 2. **Gradual migration**: dual-run while moving consumers from one system to the other. 3. **Cross-cloud / on-prem integration**: Kafka on-prem, Pub/Sub in GCP, bridge connects them. ## Trade-offs (the interview meat) - **Latency & extra hop**: the bridge is an additional network + processing stage; not ideal for ultra-low-latency hot paths. - **Ordering mismatch**: Kafka guarantees order **per partition**; Pub/Sub orders only by **ordering key** (and only when enabled). The bridge cannot perfectly preserve both models — global order is not guaranteed and key→partition mapping must be designed. - **Delivery semantics**: typically **at-least-once**, so consumers must be **idempotent**. Kafka **transactions/EOS do not survive** the bridge. - **Ack vs offset model**: Pub/Sub's per-message ack + ack-deadline redelivery differs from Kafka's offset commits; misconfiguration causes duplicates or stuck redelivery. - **Operational cost**: another component (Connect cluster or shim) to deploy, secure, scale, and monitor; you inherit **both** systems' quotas, throttling, and outage modes. - **Lock-in**: leaning on the bridge can entangle you with GCP-specific features. ## Edge cases - Large messages: Pub/Sub and Kafka have different max message sizes; the bridge may reject or split. - Backpressure: if Kafka is slow, Pub/Sub subscription backlog grows and ack deadlines may expire, causing redelivery storms. - Schema/format: you must agree on serialization (Avro/JSON/Protobuf) across both sides. ## Bottom line Great for **integration and migration**, where avoiding app rewrites matters more than squeezing out latency or EOS. For a permanent low-latency, exactly-once hot path, prefer a single native system end-to-end.

  • Why can't you rely on Kafka exactly-once semantics across a Pub/Sub-to-Kafka bridge?
    The bridge is a separate stage that re-publishes messages; Kafka's transactional producer/read_committed chain is broken, so delivery is at-least-once. Consumers must be idempotent to tolerate duplicates.
  • How do ordering guarantees differ between the two systems, and why does the bridge complicate them?
    Kafka orders per partition; Pub/Sub orders only per ordering key when enabled. The bridge must map keys to partitions, and global ordering can't be guaranteed across the hop.

saying these in an interview costs you the question

  • Claiming a bridge preserves Kafka exactly-once end to end.
  • Assuming Pub/Sub and Kafka share the same ordering model (they don't).
  • Treating the bridge as zero-cost — it's another system to run, monitor, and pay for, with combined quotas.

context

resolving links…