skip to content

What is the Confluent REST Proxy, and when would you use it to produce or consume over HTTP instead of a native Kafka client?

level: middleimportance: should knowfreq 50%

answer

  1. HTTP gateway → native Kafka protocol
  2. POST /topics to produce
  3. consumer = create/subscribe/poll/commit/DELETE
  4. Content-Type picks embedded format
  5. slower, extra hop, stateful consumers pin to a node

basics

~20 s

The Confluent REST Proxy is an HTTP service that sits in front of a Kafka cluster, letting clients produce and consume via REST/JSON instead of the binary Kafka protocol. Use it for environments that can't run a native client — restricted languages, serverless/edge, or firewalled HTTP-only networks — at the cost of higher latency and lower throughput.

solid answer

~50 s

Confluent REST Proxy is a stateless HTTP gateway (a separate process) that translates RESTful requests into native Kafka protocol calls. Producers POST records to /topics/{topic}; consumers create a server-side consumer instance, subscribe, then poll via HTTP GET, and must issue commits and a delete to release the instance. It speaks several content types (e.g. application/vnd.kafka.json.v2+json, .binary.v2+json, .avro.v2+json) and integrates with Schema Registry so Avro/Protobuf/JSON-Schema payloads are registered and validated. You use it when running a native librdkafka/JVM client isn't practical: thin/embedded/edge clients, languages without a good client, serverless functions where keeping a TCP connection and consumer-group membership is awkward, or networks that only allow HTTP. Trade-offs: HTTP overhead and per-request round trips mean lower throughput and higher latency than a native client; the proxy is an extra hop to deploy, scale, and secure; and stateful consumer instances are pinned to one proxy node, complicating load balancing. For high-volume produce/consume, prefer a native client.

go deeper

for a junior

Know it's an HTTP front-end that lets you produce/consume over REST when you can't use a native client.

for a middle

Walk through the produce POST and the stateful consumer lifecycle, and name the main trade-offs (latency, extra hop).

for a senior

Discuss content types, Schema Registry integration, consumer-node affinity, and when REST Proxy is the right architectural choice vs native.

for a principal

Weigh REST Proxy in a platform strategy: scaling/HA of the proxy tier, security boundary, governance via Schema Registry, and guiding teams away from it for high-volume paths.

**What it is.** Kafka brokers speak a custom **binary TCP wire protocol**. Any client must implement that protocol (directly or via librdkafka). The **Confluent REST Proxy** is a standalone HTTP server that you deploy alongside Kafka; it accepts ordinary **REST/JSON over HTTP(S)** and internally uses a normal Kafka client to talk to the brokers. It is the bridge for callers that can only — or prefer to — speak HTTP. **Producing.** You `POST` to `/topics/{topic}` with a JSON body containing one or more records (optionally with key, partition, and value). The proxy batches them to Kafka and returns per-record offsets/partitions (or errors). The `Content-Type` header selects the **embedded format**: `application/vnd.kafka.json.v2+json` (raw JSON values), `...binary.v2+json` (base64 bytes), or `...avro.v2+json` / `...protobuf...` / `...jsonschema...` (schema-aware, integrating with Schema Registry). **Consuming (the tricky part).** Consuming is **stateful and multi-step**: 1. `POST /consumers/{group}` to create a named **consumer instance** on the proxy; the response gives a `base_uri`. 2. `POST {base_uri}/subscription` to subscribe to topics. 3. `GET {base_uri}/records` to poll for messages (long-poll style). 4. `POST {base_uri}/offsets` (or auto-commit) to commit. 5. `DELETE {base_uri}` to destroy the instance and free resources. Because that consumer instance lives in memory on **one specific proxy node**, all subsequent requests for it must hit the same node — which complicates load balancing and means a proxy restart drops the consumer. **Schema Registry integration.** With the Avro/Protobuf/JSON-Schema content types, the proxy talks to **Confluent Schema Registry**: it registers/looks up schemas and embeds the schema **ID** so payloads are validated and consumers can deserialize. (Same wire format as native serializers — see the schema-serializer question.) **When to use it:** - Languages/runtimes lacking a solid native client. - **Thin, embedded, IoT/edge** devices that already do HTTP. - **Serverless / FaaS** (e.g. short-lived functions) where holding a persistent TCP connection and consumer-group membership is awkward — a stateless HTTP produce fits better. - **Network constraints**: environments that only permit outbound HTTP(S) through proxies/firewalls. - Quick scripts, webhooks, or integration glue. **Trade-offs (must mention):** - **Performance:** HTTP framing + JSON/base64 encoding + per-request round trips give **lower throughput and higher latency** than a native client's batched binary protocol. - **Extra component:** the proxy is another service to deploy, scale, monitor, secure (TLS, auth), and patch. - **Stateful consumers** pin to a node, limiting horizontal scaling and resilience for the consume path; produce is stateless and scales more easily. - **At-least-once nuances:** committing offsets over discrete HTTP calls requires care to avoid loss/duplication. **Rule of thumb:** REST Proxy for reach/compatibility and low-to-moderate volume; native client for high-throughput, low-latency, or stateful streaming workloads.

  • Why are REST Proxy consumers harder to load-balance than producers?
    A consumer instance is stateful and lives in memory on one proxy node (its base_uri). Every poll/commit/delete must return to that same node, so you can't freely round-robin requests; a node failure drops the consumer. Producing is stateless, so any proxy node can serve a produce request.
  • Which Content-Type would you send to produce Avro records validated by Schema Registry?
    application/vnd.kafka.avro.v2+json — the proxy then registers/looks up the schema in Schema Registry, embeds the schema ID, and validates the payload.

saying these in an interview costs you the question

  • Saying REST Proxy is faster than or replaces native clients for high throughput — it's slower due to HTTP/JSON overhead.
  • Treating REST consume as a single stateless GET — it's a stateful create→subscribe→poll→commit→delete lifecycle.
  • Claiming the proxy bypasses the Kafka protocol entirely — internally it uses a normal Kafka client.
  • Forgetting that consumer instances are pinned to one proxy node.

context

resolving links…