skip to content

KIP-932 Queues

Share groups (KIP-932) let more consumers than partitions read a topic together, locking and acknowledging records one by one. Interviewers ask when they beat a consumer group.

part ofApache Kafkaread it →
on this pageshow

explore

questions

5

What is a Kafka share group, introduced by KIP-932, and how does it differ from a Kafka consumer group?

level: juniorimportance: should knowfreq 45%

answer

  1. queues for Kafka
  2. more consumers than partitions
  3. time-limited lock per record
  4. acknowledge each record, count attempts

basics

~20 s

A share group is a Kafka group type whose members share partitions: the broker locks each record for one member, which acknowledges it individually, so the group can outgrow the partition count at the cost of ordering.

solid answer

~40 s

In a consumer group each partition has exactly one owner, so parallelism stops at the partition count and progress is one committed offset per partition. A share group, production-ready since Kafka 4.2, lets several `KafkaShareConsumer` instances read the same partition: the broker hands each record to one member under a time-limited acquisition lock, and the member settles it with `ACCEPT`, `RELEASE`, `REJECT` or `RENEW`. Delivery attempts are counted, so a record that keeps failing is archived after the delivery count limit (5 by default) instead of looping. The price is ordering: with many members on a partition and redeliveries, records are not processed in offset order. It suits independent work items; ordered streams stay on consumer groups.

go deeper

for a junior

Recall the one-line contrast: a consumer group gives each partition one owner, a share group lets many members read a partition and acknowledge records one by one.

for a middle

Explain the acquisition lock, the four acknowledgement types and the counted delivery attempts, and say what you give up in ordering.

for a senior

Show you know where the state lives (broker, __share_group_state), that delivery is at-least-once with a bounded attempt count, and that handlers must be idempotent.

for a principal

Frame the choice as ordering and offset control against elastic parallelism and per-record retry, and know the release history before betting a platform on it.

## The problem: parallelism capped by partitions In a **consumer group**, Kafka assigns each partition of a subscribed topic to exactly one member at a time. Progress is one committed offset per partition, which keeps the bookkeeping tiny and the records in offset order. It also means useful parallelism stops at the partition count: an image-thumbnail job topic with 6 partitions keeps at most 6 consumer-group members busy, and a 7th member sits idle. Adding partitions is the usual answer, but it is a topic-level change that is hard to undo, and it is a poor fit for a queue whose records are independent jobs of uneven duration. A **share group** (KIP-932, "Queues for Kafka") is a separate kind of group that exists alongside consumer groups and is built for that workload. Its members are **share consumers** — `KafkaShareConsumer` in the Java client — and they cooperatively consume the same partitions. The design notes describe four fundamental differences: partitions may be assigned to several consumers, the number of consumers may exceed the number of partitions, records are acknowledged individually, and delivery attempts are counted. ## How a share group works - **Partitions are shared, and the broker assigns them.** The group coordinator runs a server-side share-group assignor (the built-in simple assignor, chosen by the broker config `group.share.assignors`) and may give one partition to several members. There is no client-side leader; the share consumer refuses `partition.assignment.strategy` and `group.remote.assignor` with a `ConfigException`. - **Membership is kept by the broker too.** Members join and stay alive through the `ShareGroupHeartbeat` request. Session timeout and heartbeat interval are group settings (`share.session.timeout.ms`, default 45 s; `share.heartbeat.interval.ms`, default 5 s), and the client rejects `session.timeout.ms` and `heartbeat.interval.ms`. - **Records are locked, not owned.** When a member fetches, the broker *acquires* each delivered record for that member under a time-limited **acquisition lock**, 30 seconds by default. While it is held, no other member of that share group receives the record. - **Records are settled one by one.** The member acknowledges each record with an `AcknowledgeType`: `ACCEPT` (processed), `RELEASE` (deliver it again), `REJECT` (unprocessable, never deliver again) or `RENEW` (still working, extend the lock). Doing nothing lets the lock expire, and the record becomes available again, within the limit below. - **Delivery attempts are counted.** Each acquisition is one attempt. Once a record has used up the delivery count limit (broker default `group.share.delivery.count.limit=5`), it is archived instead of being made available again, so a poison record cannot loop forever. - **State lives on the broker.** Each share-partition's start offset and the state of its in-flight records are persisted by the share coordinator in the internal `__share_group_state` topic. ## Consumer group and share group side by side | | Consumer group | Share group | |---|---|---| | Partition ownership | one member per partition | several members per partition | | Useful members | up to the partition count | can exceed the partition count | | Progress record | committed offset per partition | per-record state held by the broker | | Acknowledgement | offset commit covering all earlier records | per record: ACCEPT, RELEASE, REJECT, RENEW | | Failing record | the application decides; the offset waits or skips | redelivered, then archived at the delivery limit | | Ordering | offset order within a partition | not preserved across members or redeliveries | | Java client | `KafkaConsumer` | `KafkaShareConsumer` | ## What you give up 1. **Processing order.** One poll returns a partition's records in increasing offset order, but other members are processing neighbouring records at the same time and a released record comes back later, so the application must not depend on order. 2. **Offset-level control in the client.** A share consumer has no `seek()` or `assign()`. Where a new share group starts is the group config `share.auto.offset.reset`, default `latest`, and moving it is an operator action with `kafka-share-groups.sh --reset-offsets` while the group has no active members. 3. **Exactly-once processing.** A record whose lock expires after the work is done but before the acknowledgement arrives is delivered again, so handlers must be idempotent. Treat delivery as at-least-once with a bounded number of attempts. ## Release status Share groups were early access in Kafka 4.0, a preview in 4.1 (enabled by upgrading the `share.version` feature to 1 with `kafka-features.sh`) and production-ready from 4.2. A topic can be read at the same time, independently, by any mix of consumer groups and share groups; each group keeps its own progress.

  • In a Kafka share group, who decides which members read which partitions?
    The broker's group coordinator. It runs a server-side share-group assignor (the built-in simple assignor, set by `group.share.assignors`) and delivers the assignment through `ShareGroupHeartbeat`, and it may give one partition to several members. There is no client-side leader: a `KafkaShareConsumer` rejects `partition.assignment.strategy` and `group.remote.assignor`, and its session timeout and heartbeat interval are group configs rather than client configs.
  • Can a consumer group and a share group read the same Kafka topic at the same time?
    Yes. Each group consumes the topic independently and keeps its own progress: the consumer group its committed offsets, the share group its per-record state in `__share_group_state`. Acknowledging a record in the share group does not delete it or hide it from anyone else; the log keeps it until normal retention removes it.

  • Extra share-group members beyond the partition count sit idle, as in a consumer group.
  • A share group keeps per-partition processing order like a consumer group does.
  • Acknowledging a record in a share group deletes it from the topic.
  • Kafka still has no per-record acknowledgement, only offset commits.
  • A share group is just a consumer group with a different assignor.

Assumes Apache Kafka 4.2 or later; share groups were early access in 4.0, a preview in 4.1 and production-ready in 4.2.

open as a page

In a KafkaShareConsumer, how do implicit and explicit acknowledgement differ, and how does share.acknowledgement.mode change the poll loop?

level: middleimportance: should knowfreq 30%

basics

~10 s

With share.acknowledgement.mode=implicit, the default, the next poll() or commit accepts the whole previous batch; with explicit, the application calls acknowledge(record, type) for every record before polling again, which allows per-record RELEASE, REJECT and RENEW.

open as a page

In a Kafka share group, how do acquisition locks and the AcknowledgeType values ACCEPT, RELEASE, REJECT and RENEW decide a record's fate?

level: middleimportance: should knowfreq 38%

basics

~20 s

Fetching acquires each record for one share consumer under a lock (30 s by default); ACCEPT finishes it, RELEASE returns it for another attempt, REJECT archives it, RENEW extends the lock, and an expired lock returns it automatically.

open as a page

A Kafka share group runs 40 thumbnail workers on a 6-partition topic, and one corrupt image keeps crashing workers; how do delivery counting and locks contain it, and what would you change?

level: seniorimportance: should knowfreq 28%

basics

~20 s

Each crash leaves the image locked until the lock expires or the broker frees it, then it is redelivered; at the delivery count limit (5 by default) it is archived. Catching the error and acknowledging REJECT archives it at once.

open as a page

When should a team run a Kafka workload on a share group instead of a consumer group or a separate queue broker, and what does adopting share groups cost?

level: principalimportance: nice to knowfreq 24%

basics

~20 s

Use a share group for independent jobs that need more workers than partitions or per-record retry; keep a consumer group for ordering, keyed state or exactly-once; and budget a 4.2+ cluster, share.version, a client rewrite and new tooling.

open as a page