What is a Kafka share group, introduced by KIP-932, and how does it differ from a Kafka consumer group?
answer
- queues for Kafka
- more consumers than partitions
- time-limited lock per record
- acknowledge each record, count attempts
basics
~20 sA share group is a Kafka group type whose members share partitions: the broker locks each record for one member, which acknowledges it individually, so the group can outgrow the partition count at the cost of ordering.
solid answer
~40 sIn a consumer group each partition has exactly one owner, so parallelism stops at the partition count and progress is one committed offset per partition. A share group, production-ready since Kafka 4.2, lets several `KafkaShareConsumer` instances read the same partition: the broker hands each record to one member under a time-limited acquisition lock, and the member settles it with `ACCEPT`, `RELEASE`, `REJECT` or `RENEW`. Delivery attempts are counted, so a record that keeps failing is archived after the delivery count limit (5 by default) instead of looping. The price is ordering: with many members on a partition and redeliveries, records are not processed in offset order. It suits independent work items; ordered streams stay on consumer groups.
go deeper
Recall the one-line contrast: a consumer group gives each partition one owner, a share group lets many members read a partition and acknowledge records one by one.
Explain the acquisition lock, the four acknowledgement types and the counted delivery attempts, and say what you give up in ordering.
Show you know where the state lives (broker, __share_group_state), that delivery is at-least-once with a bounded attempt count, and that handlers must be idempotent.
Frame the choice as ordering and offset control against elastic parallelism and per-record retry, and know the release history before betting a platform on it.
## The problem: parallelism capped by partitions In a **consumer group**, Kafka assigns each partition of a subscribed topic to exactly one member at a time. Progress is one committed offset per partition, which keeps the bookkeeping tiny and the records in offset order. It also means useful parallelism stops at the partition count: an image-thumbnail job topic with 6 partitions keeps at most 6 consumer-group members busy, and a 7th member sits idle. Adding partitions is the usual answer, but it is a topic-level change that is hard to undo, and it is a poor fit for a queue whose records are independent jobs of uneven duration. A **share group** (KIP-932, "Queues for Kafka") is a separate kind of group that exists alongside consumer groups and is built for that workload. Its members are **share consumers** — `KafkaShareConsumer` in the Java client — and they cooperatively consume the same partitions. The design notes describe four fundamental differences: partitions may be assigned to several consumers, the number of consumers may exceed the number of partitions, records are acknowledged individually, and delivery attempts are counted. ## How a share group works - **Partitions are shared, and the broker assigns them.** The group coordinator runs a server-side share-group assignor (the built-in simple assignor, chosen by the broker config `group.share.assignors`) and may give one partition to several members. There is no client-side leader; the share consumer refuses `partition.assignment.strategy` and `group.remote.assignor` with a `ConfigException`. - **Membership is kept by the broker too.** Members join and stay alive through the `ShareGroupHeartbeat` request. Session timeout and heartbeat interval are group settings (`share.session.timeout.ms`, default 45 s; `share.heartbeat.interval.ms`, default 5 s), and the client rejects `session.timeout.ms` and `heartbeat.interval.ms`. - **Records are locked, not owned.** When a member fetches, the broker *acquires* each delivered record for that member under a time-limited **acquisition lock**, 30 seconds by default. While it is held, no other member of that share group receives the record. - **Records are settled one by one.** The member acknowledges each record with an `AcknowledgeType`: `ACCEPT` (processed), `RELEASE` (deliver it again), `REJECT` (unprocessable, never deliver again) or `RENEW` (still working, extend the lock). Doing nothing lets the lock expire, and the record becomes available again, within the limit below. - **Delivery attempts are counted.** Each acquisition is one attempt. Once a record has used up the delivery count limit (broker default `group.share.delivery.count.limit=5`), it is archived instead of being made available again, so a poison record cannot loop forever. - **State lives on the broker.** Each share-partition's start offset and the state of its in-flight records are persisted by the share coordinator in the internal `__share_group_state` topic. ## Consumer group and share group side by side | | Consumer group | Share group | |---|---|---| | Partition ownership | one member per partition | several members per partition | | Useful members | up to the partition count | can exceed the partition count | | Progress record | committed offset per partition | per-record state held by the broker | | Acknowledgement | offset commit covering all earlier records | per record: ACCEPT, RELEASE, REJECT, RENEW | | Failing record | the application decides; the offset waits or skips | redelivered, then archived at the delivery limit | | Ordering | offset order within a partition | not preserved across members or redeliveries | | Java client | `KafkaConsumer` | `KafkaShareConsumer` | ## What you give up 1. **Processing order.** One poll returns a partition's records in increasing offset order, but other members are processing neighbouring records at the same time and a released record comes back later, so the application must not depend on order. 2. **Offset-level control in the client.** A share consumer has no `seek()` or `assign()`. Where a new share group starts is the group config `share.auto.offset.reset`, default `latest`, and moving it is an operator action with `kafka-share-groups.sh --reset-offsets` while the group has no active members. 3. **Exactly-once processing.** A record whose lock expires after the work is done but before the acknowledgement arrives is delivered again, so handlers must be idempotent. Treat delivery as at-least-once with a bounded number of attempts. ## Release status Share groups were early access in Kafka 4.0, a preview in 4.1 (enabled by upgrading the `share.version` feature to 1 with `kafka-features.sh`) and production-ready from 4.2. A topic can be read at the same time, independently, by any mix of consumer groups and share groups; each group keeps its own progress.
- In a Kafka share group, who decides which members read which partitions?The broker's group coordinator. It runs a server-side share-group assignor (the built-in simple assignor, set by `group.share.assignors`) and delivers the assignment through `ShareGroupHeartbeat`, and it may give one partition to several members. There is no client-side leader: a `KafkaShareConsumer` rejects `partition.assignment.strategy` and `group.remote.assignor`, and its session timeout and heartbeat interval are group configs rather than client configs.
- Can a consumer group and a share group read the same Kafka topic at the same time?Yes. Each group consumes the topic independently and keeps its own progress: the consumer group its committed offsets, the share group its per-record state in `__share_group_state`. Acknowledging a record in the share group does not delete it or hide it from anyone else; the log keeps it until normal retention removes it.
- Extra share-group members beyond the partition count sit idle, as in a consumer group.
- A share group keeps per-partition processing order like a consumer group does.
- Acknowledging a record in a share group deletes it from the topic.
- Kafka still has no per-record acknowledgement, only offset commits.
- A share group is just a consumer group with a different assignor.
Assumes Apache Kafka 4.2 or later; share groups were early access in 4.0, a preview in 4.1 and production-ready in 4.2.