How does Kafka's wire-protocol API versioning let clients of different versions and languages interoperate with brokers, and what is the role of ApiVersions negotiation?

**Why versioning exists.** Kafka clients and brokers are upgraded independently, on different schedules, in many languages. Without a negotiation mechanism, any version mismatch would break communication. Kafka solves this with **fine-grained, per-request-type protocol versioning** plus a runtime negotiation handshake. **The protocol shape.** Kafka's wire protocol is a catalog of **request/response message types**, each with: - an **API key** — a stable integer identifying the request type. Examples: `Produce=0`, `Fetch=1`, `ListOffsets=2`, `Metadata=3`, `OffsetCommit=8`, `OffsetFetch=9`, `FindCoordinator=10`, `JoinGroup=11`, `ApiVersions=18`, and so on. - an **API version** — a small integer that **increments independently per API key** whenever that request/response's schema changes (a new field, a changed encoding, a new behavior). So 'Fetch v12' and 'Produce v9' are independent; bumping one never bumps the other. **The ApiVersions handshake (KIP-35).** When a client establishes a connection (after SASL/TLS), it sends an **ApiVersions request** (API key 18). The broker replies with, **for every API key it supports**, a `[min_version, max_version]` pair. The client then, for each request type it wants to use, computes the **intersection** of the broker's supported range and its own supported range and selects the **highest mutually supported version**. If there's no overlap for a required API, the client errors out (UNSUPPORTED_VERSION). This is the linchpin of interoperability: - A **newer client** can talk to an **older broker** by downgrading individual requests to versions the broker understands. - An **older client** works with a **newer broker** because brokers retain support for old request versions. - **librdkafka and the JVM client** interoperate with the same brokers despite being separate implementations, because each independently negotiates against the broker's advertised capabilities rather than hardcoding a global protocol version. **Evolving the encoding: flexible/tagged fields (KIP-482).** Originally, adding a field meant a hard version bump and rigid parsing. **Flexible versions** introduced **optional tagged fields** and compact (varint-length) encodings, so brokers/clients can add optional fields that older peers simply skip — improving forward compatibility and reducing version churn. A given API key has a 'flexible version' threshold above which the flexible encoding applies. **Operational consequences (the principal-level payoff):** - **Rolling upgrades:** the supported pattern is **upgrade brokers first**, then clients. Brokers on a newer version still accept old request versions, so existing clients keep working during the broker roll; afterwards clients can be upgraded to use newer request versions/features. - **inter.broker.protocol.version (IBP):** controls the protocol version brokers use to talk to **each other**; you typically only advance it after all brokers are upgraded, then perform a second rolling restart. This gates when cluster-internal new features 'switch on.' - **Message format / down-conversion:** if clients request an older fetch version than the on-disk message format, the broker may **down-convert** messages (CPU/heap cost, loses zero-copy). Keeping clients and `log.message.format` aligned avoids this tax. - **Capability gating:** a client only uses a feature if the negotiated version supports it — e.g. idempotent/transactional produce, incremental fetch sessions (KIP-227), or the new consumer group protocol (KIP-848) — which ties back to JVM-vs-librdkafka parity: a feature requires both the right client AND a broker advertising the needed API version. **Diagnostics.** The `kafka-broker-api-versions.sh` CLI dumps each broker's supported `[min,max]` per API key — invaluable when debugging UNSUPPORTED_VERSION errors or planning an upgrade. UNSUPPORTED_VERSION almost always means a client is asking for a request version the broker (or a feature flag) doesn't support yet. **Bottom line.** Per-API independent versioning + the ApiVersions handshake + flexible fields are what make Kafka's enormous, multi-language, multi-version client ecosystem hang together. Clients negotiate capabilities at connect time instead of assuming a monolithic version, which is precisely why cross-version and cross-language interoperability is robust.