skip to content

Compare the two ways to get the result of a send: the returned Future versus a Callback. When is each appropriate, and what are the pitfalls?

level: middleimportance: should knowfreq 58%

answer

  1. Future.get() blocks → synchronous, kills batching
  2. Callback async on Sender thread → non-blocking
  3. RecordMetadata: partition/offset/timestamp
  4. exactly one of (metadata, exception) non-null
  5. don't block the Sender thread in a callback

basics

~20 s

send() returns a Future<RecordMetadata>; calling future.get() blocks until the broker responds, turning async into sync. Alternatively pass a Callback to send(record, callback) that the producer invokes asynchronously on completion. Use Future.get() for sync confirmation, Callback for non-blocking handling.

solid answer

~50 s

Both report the same outcome — success (RecordMetadata: partition, offset, timestamp) or failure (an Exception). The Future is returned by every send(); future.get() blocks the caller until completion, effectively making the send synchronous (low throughput, simple error handling). A Callback (onCompletion(RecordMetadata, Exception)) passed to send(record, callback) is invoked by the Sender thread when the record is acked or finally fails — non-blocking, preserves async throughput. Key pitfalls: (1) Callbacks run on the single Sender thread, so any blocking/slow work there stalls ALL sends; keep them fast. (2) Per-partition callback ordering is guaranteed, but cross-partition ordering is not. (3) Exactly one of RecordMetadata or Exception is non-null. (4) Calling future.get() right after each send serializes the pipeline and kills batching. Use Callback (or future.get() in a separate stage / via a queue) for high throughput; use future.get() only when you genuinely need synchronous confirmation.

code

java · 17 lines
java
// Async with Callback (preserves batching/throughput)
producer.send(record, (RecordMetadata md, Exception e) -> {
    if (e != null) {
        // exactly one of md / e is non-null here
        log.error("send failed for key={} : {}", key, e.toString());
    } else {
        log.debug("sent to {}-{} @ offset {}", md.topic(), md.partition(), md.offset());
    }
    // keep this fast: it runs on the Sender thread
});

// Synchronous confirmation (blocks; kills batching if done per-record)
try {
    RecordMetadata md = producer.send(record).get();
} catch (ExecutionException ex) {
    throw new RuntimeException(ex.getCause()); // real cause is wrapped
}

go deeper

for a junior

Know send() returns a Future and you can also pass a Callback; future.get() waits for the result.

for a middle

Explain the throughput cost of future.get(), the Callback signature, and that exactly one of metadata/exception is set.

for a senior

Discuss callback-thread constraints, per-partition ordering, retriable vs non-retriable errors in the callback.

for a principal

Set patterns for the team: when sync confirmation is required vs callback pipelines, offloading heavy callback work, and interaction with idempotence/ordering.

## The two completion mechanisms Every `send()` returns a **`Future<RecordMetadata>`**. There is also an overload **`send(record, Callback)`**. Both deliver the *same* result, just at different times and on different threads. ### RecordMetadata (success payload) On success the producer hands back a `RecordMetadata` with the **topic-partition**, the **offset** the record landed at, and its **timestamp** (plus serialized sizes). This is your proof of where the record was stored. ### Future - `Future<RecordMetadata> f = producer.send(record);` - `f.get()` **blocks the calling thread** until the Sender completes the record (broker ack per `acks`, or final failure → `ExecutionException` wrapping the cause). - Calling `f.get()` immediately after each send makes producing **synchronous**: each record round-trips before the next, destroying batching and throughput. It's simple and gives in-line error handling, fine for low-rate or strict-confirmation cases. ### Callback - `producer.send(record, (metadata, exception) -> { ... });` - The producer invokes `onCompletion(RecordMetadata, Exception)` **asynchronously, on the Sender thread**, when the record is acked or finally fails. - **Exactly one** of the two args is non-null: success → `metadata` set, `exception` null; failure → `exception` set, `metadata` (mostly) null. - Non-blocking: the caller keeps producing, so batching and throughput are preserved. ## Ordering guarantees Callbacks for records sent to the **same partition** are invoked **in send order**. Across partitions there is **no ordering guarantee**. (With `enable.idempotence=true` and bounded in-flight requests, per-partition ordering is preserved even across retries.) ## Pitfalls 1. **Don't block the Sender thread.** Callbacks run on the single producer I/O thread. Doing slow work (DB calls, another blocking send, heavy logging) inside a callback stalls *every* partition's progress. Offload heavy work to another executor. 2. **Don't `future.get()` per record** in the hot path — it serializes the pipeline. If you must confirm synchronously in bulk, send many, then get() them, or use callbacks + a latch. 3. **Handle both error classes.** Some exceptions are retriable (the producer already retried) and arrive only after retries exhaust; others (e.g. `RecordTooLargeException`, serialization errors) are non-retriable. Your callback must distinguish and react. 4. **Closures capture cost.** A callback per record allocates; at very high rates reuse/structure to limit garbage. ## When to use which - **Future.get()** — synchronous confirmation, simple scripts, tests, or a request that must not proceed until the record is durably acked. - **Callback** — production high-throughput pipelines where you want delivery confirmation/error handling without blocking.

  • Why is calling future.get() immediately after every send() bad for throughput?
    It blocks the caller until each record is acked before sending the next, so records can't accumulate into batches. You lose batching/compression and reduce to one in-flight record at a time — effectively synchronous, low-throughput producing.
  • What ordering can you rely on for callback invocation?
    Callbacks for records to the same partition fire in the order the records were sent. There's no cross-partition ordering guarantee. Idempotence with bounded in-flight requests preserves per-partition order even across retries.

saying these in an interview costs you the question

  • Saying both RecordMetadata and Exception can be set at once — exactly one is non-null.
  • Doing blocking work inside a callback — it stalls the shared Sender thread.
  • Claiming callbacks run on the application thread — they run on the Sender thread.
  • Using future.get() per record and still expecting high throughput.

context

resolving links…