How does Kafka's bidirectional client/broker compatibility work, and what does it mean for upgrading clients vs. brokers?
answer
- ApiVersions request on connect
- per-API min/max, pick highest common
- KIP-35 / 0.10.2 => bidirectional
- upgrade brokers & clients in either order
- old client + new format => down-conversion penalty
basics
~20 sEach Kafka API (request type) is versioned. On connect, the client asks the broker which versions it supports (ApiVersions) and uses the highest both understand. Since ~0.10.2 this works both ways: newer clients talk to older brokers and older clients talk to newer brokers. Upgrade brokers and clients independently.
solid answer
~50 sKafka's protocol is a set of independently versioned request APIs (Produce, Fetch, Metadata, etc.). When a client connects it sends an **ApiVersions** request; the broker replies with the min/max version it supports for each API. The client then picks, per API, the highest version both sides support and frames its requests accordingly. Historically (pre-0.10.2) only *backward* compatibility held — new brokers served old clients — so you always upgraded brokers first. Since **KIP-35 / 0.10.2**, compatibility is **bidirectional**: a newer client can talk to an older broker (it downshifts to versions the broker knows) and an older client keeps working against a newer broker. Practically this means you can upgrade brokers and clients on independent schedules, in either order, without a flag-day. Caveats: a client can only use features the broker actually supports, and `log.message.format.version` / message-format down-conversion can still penalize very old clients against newer brokers.
go deeper
Know that clients and brokers negotiate versions and that they can be upgraded independently on modern Kafka.
Explain ApiVersions, per-API version selection, and the bidirectional guarantee since 0.10.2 (KIP-35).
Reason about feature gating, down-conversion cost, and how to debug interop with kafka-broker-api-versions.sh.
Set fleet-wide upgrade policy that exploits independent client/broker rollout while managing format down-conversion and feature-gating risk.
**The protocol is versioned per API.** Kafka's wire protocol is not one monolithic version. Each *request type* — Produce, Fetch, Metadata, ListOffsets, JoinGroup, and dozens more — is independently versioned (v0, v1, v2, …). New features arrive as new versions of specific APIs. **ApiVersions negotiation.** When a client opens a connection it first issues an **ApiVersions** request. The broker responds with, for every API key it implements, the **minimum and maximum version** it supports. The client intersects that with its own supported range and, per API, chooses the **highest mutually-supported version**. So negotiation is fine-grained and per-API, not a single global handshake number. **Backward vs. bidirectional compatibility.** - *Backward compatibility* (always true): a newer **broker** continues to serve requests at older API versions, so **older clients keep working** against upgraded brokers. This is why the historical rule was 'upgrade brokers before clients.' - *Bidirectional compatibility* (since **KIP-35**, Kafka **0.10.2**): clients can *discover* broker capabilities via ApiVersions and **downshift**, so a **newer client also works against an older broker** by using only versions the broker advertises. Combined, you get full bidirectional support: brokers and clients can be upgraded in **either order, independently**. **What this means operationally.** You no longer need a flag-day. You can roll out a new client library while brokers are still on the old version (the client uses older API versions where needed), or upgrade brokers first and let old clients keep running. This decoupling is what makes large fleets manageable — thousands of producer/consumer apps don't have to be redeployed in lockstep with a broker upgrade. **Limits and caveats.** 1. **Feature availability is gated by the broker.** A new client cannot use an API version the broker doesn't support; that feature is simply unavailable until the broker is upgraded (and, for some features, until `inter.broker.protocol.version`/`metadata.version` is raised). 2. **Message format down-conversion.** If brokers store records in a newer `log.message.format.version` than a very old client understands, the broker must *down-convert* records on read, which disables the zero-copy send optimization and adds CPU/latency. So 'old client + new broker' works but can be slow if formats diverge. 3. **Very old clients.** Compatibility guarantees are strong but not infinite; extremely old clients predating ApiVersions (pre-0.10.0) have weaker guarantees. 4. **Check ApiVersions to debug.** `kafka-broker-api-versions.sh` (or a client's debug logs) shows exactly what each broker advertises — invaluable when a feature 'isn't working' because the broker is too old. **Mental model.** Think of it as each side declaring 'I speak versions X–Y of each dialect'; they always converge on the highest dialect both know. That handshake, not a single cluster version, is what governs client/broker interop.
- Before KIP-35, what was the mandatory upgrade order and why?Brokers before clients. Only backward compatibility held — new brokers served old API versions — but old brokers could not serve a newer client's requests, so clients had to wait until brokers supported the versions they used.
- An old consumer is unusually slow against a newer broker. What protocol-level cause should you suspect?Message-format down-conversion: the broker stores records in a newer log.message.format.version than the client understands, so it must rewrite records on read, losing the zero-copy optimization and adding CPU/latency.
saying these in an interview costs you the question
- Saying clients and brokers share a single global protocol version number.
- Claiming you must always upgrade brokers before clients (true only pre-0.10.2).
- Asserting a new client can use new features against an old broker.
- Ignoring down-conversion cost for old clients on newer brokers.