skip to content

In a KafkaShareConsumer, how do implicit and explicit acknowledgement differ, and how does share.acknowledgement.mode change the poll loop?

level: middleimportance: should knowfreq 30%

answer

  1. default is implicit
  2. next poll accepts the batch
  3. explicit: every record, every poll
  4. IllegalStateException both ways

basics

~10 s

With share.acknowledgement.mode=implicit, the default, the next poll() or commit accepts the whole previous batch; with explicit, the application calls acknowledge(record, type) for every record before polling again, which allows per-record RELEASE, REJECT and RENEW.

solid answer

~30 s

`share.acknowledgement.mode` defaults to `implicit`: you never call `acknowledge()` (it throws `IllegalStateException`), and the next `poll()` — or `commitSync()`/`commitAsync()` — marks every record from the previous poll as `ACCEPT`. `close()` releases the outstanding records instead, so commit first. With `explicit`, you call `acknowledge(record, AcknowledgeType.X)` on every record before the next `poll()`, or that `poll()` throws `IllegalStateException`. The calls are local; they reach the broker on the next `poll()`, a commit, or `close()`. Implicit fits batches that succeed or fail together; explicit is needed once a record can be released, rejected or renewed on its own.

code

java · 25 lines
java
Properties props = new Properties();
props.setProperty("bootstrap.servers", "localhost:9092");
props.setProperty("group.id", "thumbnails");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
props.setProperty("share.acknowledgement.mode", "explicit");

try (KafkaShareConsumer<String, byte[]> consumer = new KafkaShareConsumer<>(props)) {
    consumer.subscribe(List.of("image-jobs"));
    while (running) {
        ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofMillis(1000));
        for (ConsumerRecord<String, byte[]> record : records) {
            try {
                makeThumbnail(record.value());
                consumer.acknowledge(record, AcknowledgeType.ACCEPT);
            } catch (CorruptImageException e) {
                consumer.acknowledge(record, AcknowledgeType.REJECT);
            } catch (Exception e) {
                consumer.acknowledge(record, AcknowledgeType.RELEASE);
            }
        }
        Map<TopicIdPartition, Optional<KafkaException>> result = consumer.commitSync();
        result.forEach((tp, error) -> error.ifPresent(e -> log.warn("ack failed for {}", tp, e)));
    }
}

go deeper

for a junior

Know the config name, that implicit is the default, and that explicit means calling acknowledge() on each record.

for a middle

Explain when acknowledgements are actually sent, what poll() and close() do in each mode, and which IllegalStateException each mode throws.

for a senior

Spot the silent-accept bug in an implicit loop that swallows exceptions, and wire commitSync results or a commit callback into error handling.

for a principal

Decide a team default: implicit for all-or-nothing batch jobs, explicit wherever records fail individually, enforced in a shared consumer template.

## Two ways to tell the broker a record is done A `KafkaShareConsumer` is the Java client for a Kafka **share group**, a group whose members share partitions and acknowledge records one by one. Every record it receives from `poll()` is held under an acquisition lock until the consumer acknowledges it. The client config `share.acknowledgement.mode` chooses how that happens, and its default is `implicit`. ## Implicit acknowledgement (the default) In implicit mode the application never calls `acknowledge()`; in fact `acknowledge()` throws `IllegalStateException` ("Implicit acknowledgement of delivery is being used"). Acknowledgement happens as a side effect of the next call: - **The next `poll()`** marks every record delivered by the previous poll as `ACCEPT` and commits those acknowledgements asynchronously. A failure to commit them raises no exception. - **`commitSync()` or `commitAsync()`** marks all delivered records as `ACCEPT` and commits them. This costs one extra request but lets you see the outcome; `commitSync()` returns a `Map<TopicIdPartition, Optional<KafkaException>>`. - **`close()`** does not accept the outstanding batch: it releases any acquired records without acknowledgement, so they are delivered again. Commit before closing if the last batch was processed. Implicit mode is all-or-nothing per batch. If processing one record throws and the loop simply carries on to the next `poll()`, that failed record is accepted with the rest. To get it retried, the batch must not be accepted — the process dies or closes without committing, so the whole batch is redelivered — or the consumer switches to explicit mode. ## Explicit acknowledgement With `share.acknowledgement.mode=explicit` the application must acknowledge **every** record returned by `poll()` with `acknowledge(record, type)` — `ACCEPT`, `RELEASE`, `REJECT` or `RENEW` — before it polls again. Calling `poll()` with records still unacknowledged throws `IllegalStateException`; the remaining records can still be acknowledged afterwards. 1. `acknowledge()` only updates local state in the consumer; nothing is sent yet. 2. The acknowledgements reach the broker on the next `poll()` (asynchronously, no exception on failure), on `commitSync()` / `commitAsync()`, or on `close()`, which tries to commit pending acknowledgements and releases whatever is still acquired. 3. For each topic-partition, the acknowledgements of a batch are applied atomically, so there is one error per partition to handle. 4. An `AcknowledgementCommitCallback`, registered with `setAcknowledgementCommitCallback()`, reports the outcome of asynchronous commits. ## What the broker sees - Both modes end with the same broker-side outcomes per record: acknowledged, available again, archived, or still acquired under a renewed lock. - Implicit mode can only express `ACCEPT`; records come back to the group only when a lock expires or `close()` releases them. Explicit mode can express all four types. - Acknowledgements committed by `poll()` travel on the share-fetch request itself. `commitSync()` and `commitAsync()` send an additional request, which the client documentation calls slightly less efficient, in exchange for a visible result. ## How to choose | Situation | Mode | |---|---| | Every record either succeeds or the process dies | implicit | | Some records fail permanently and should not be retried | explicit, `REJECT` | | Transient failures should be retried by any member | explicit, `RELEASE` | | Processing can outlast the lock | explicit, `RENEW` | | You need per-record failure handling at all | explicit | The rule of thumb: implicit is the shortest loop and is correct when a batch is effectively all-or-nothing; explicit is required as soon as one record's outcome can differ from its neighbours'. ## Things that trip people up - **Mixing the APIs.** A share consumer has `subscribe()`, `poll()`, `acknowledge()`, `commitSync()` and `commitAsync()`, but no `seek()`, `assign()` or offset-taking commit. Code ported from `KafkaConsumer` that calls those methods does not compile. - **Copying consumer-group configs.** The share consumer rejects `enable.auto.commit`, `auto.offset.reset`, `isolation.level`, `group.instance.id`, `session.timeout.ms` and `heartbeat.interval.ms` with a `ConfigException`; the equivalents that exist are group configs on the broker side. - **The mode is fixed per consumer instance.** It is read once from the config when the consumer is built. - **Batch size.** In the default `share.acquire.mode=batch_optimized` a poll may return more than `max.poll.records`, aligned to stored batches; `record_limit` caps it at `max.poll.records` and stops prefetching, which matters when each record is slow.

  • What happens in implicit mode when processing one record of a batch throws and the loop continues?
    The next `poll()` accepts the whole previous batch, including the record that failed, so its failure is lost. Implicit mode has no way to mark one record of a batch as failed. Either let the process die so the locks expire and the batch is redelivered, or switch to explicit mode and `RELEASE` or `REJECT` that record.
  • How do you find out that an acknowledgement commit failed in a share consumer?
    `commitSync()` returns a map from each `TopicIdPartition` to an optional `KafkaException`, one result per partition because a batch's acknowledgements are applied atomically per partition. For asynchronous commits — `commitAsync()` or the commit piggybacked on `poll()` — register an `AcknowledgementCommitCallback` with `setAcknowledgementCommitCallback()`; the piggybacked commit raises no exception itself.

  • In implicit mode you still call acknowledge() for failed records.
  • Explicit mode sends each acknowledge() call to the broker immediately.
  • Unacknowledged records in explicit mode are accepted on the next poll().
  • close() in implicit mode accepts the last batch it received.
  • A share consumer commits offsets with commitSync(offsets) like KafkaConsumer.