skip to content

Consumers and Consumer Groups

The read path: the poll loop, consumer groups and rebalancing, offset commits, and lag. Interviewers spend most of their Kafka time here because nearly every production issue surfaces as a consumer symptom.

part ofApache Kafkaread it →
on this pageshow

explore

questions

65 · 13 sections

What does KafkaConsumer.poll(Duration) actually do, and why must you call it in a continuous loop?

level: juniorimportance: must knowfreq 80%
basics
~20 s

poll(Duration) fetches a batch of records the consumer already has buffered (or waits up to the timeout for some), and also drives heartbeats, group coordination, and offset commits. You loop because all consumer work happens during poll.

open as a page

What does max.poll.records control, and how does it differ from the fetch-size settings?

level: middleimportance: must knowfreq 70%
basics
~20 s

max.poll.records caps how many records a single poll() call returns to your code (default 500). Fetch-size settings (fetch.max.bytes, max.partition.fetch.bytes) control how much data is pulled from brokers over the wire — bytes, not record count.

open as a page

Explain how fetch.min.bytes and fetch.max.wait.ms work together to trade off latency vs throughput in consumer fetches.

level: middleimportance: should knowfreq 55%
basics
~20 s

fetch.min.bytes tells the broker the minimum data to accumulate before answering a fetch; the broker waits up to fetch.max.wait.ms for that much to arrive, then responds anyway. Bigger min.bytes = better throughput but higher latency.

open as a page

What is max.partition.fetch.bytes, how does it relate to fetch.max.bytes, and what failure mode arises if a record exceeds it?

level: seniorimportance: should knowfreq 45%
basics
~20 s

max.partition.fetch.bytes caps bytes returned per partition per fetch (default ~1MB); fetch.max.bytes caps total bytes across all partitions (~50MB). Both are soft limits: to avoid stalling, a broker always returns at least one full record batch even if it exceeds the cap.

open as a page

Describe how the consumer's per-partition fetch buffers and prefetch pipelining work to keep poll() fast and overlap network with processing.

level: seniorimportance: should knowfreq 35%
basics
~20 s

The consumer keeps a buffer of records per partition and issues fetch requests for more data BEFORE the buffer is empty (prefetch). So while you process the current batch, the next fetch is already in flight, and poll() usually returns instantly from the buffer instead of waiting on the network.

open as a page

What is the Group Coordinator in Kafka, and how is the coordinator broker for a particular consumer group chosen?

level: juniorimportance: must knowfreq 70%
basics
~10 s

The Group Coordinator is a broker that manages a consumer group: it handles members joining/leaving, triggers rebalances, and stores offsets. The coordinator is the broker that leads the __consumer_offsets partition the group maps to.

open as a page

Walk through the JoinGroup and SyncGroup phases of a rebalance. Who computes the partition assignment, and how does it reach each member?

level: middleimportance: must knowfreq 65%
basics
~20 s

All members send JoinGroup; the coordinator picks one member as group leader and returns the full member list to it. The leader computes assignments and sends them in SyncGroup; the coordinator fans them out to each member in their SyncGroup response.

open as a page

Describe the consumer group states the coordinator maintains (Empty, PreparingRebalance, CompletingRebalance, Stable, Dead) and the transitions between them.

level: middleimportance: should knowfreq 40%
basics
~10 s

The coordinator tracks a group's lifecycle: Empty (no members), PreparingRebalance (waiting for members to join), CompletingRebalance (awaiting SyncGroup), Stable (assignment done, consuming), and Dead (group removed). Membership changes drive transitions through these states.

open as a page

What are the generation ID and member.id in the consumer group protocol, and how do they protect against stale/zombie members?

level: seniorimportance: should knowfreq 45%
basics
~20 s

The generation ID is a monotonically increasing counter the coordinator bumps on each successful rebalance; member.id uniquely identifies a member within the group. Requests carrying a stale generation or unknown member.id are rejected, fencing zombies.

open as a page

Why does the replication factor and configuration of the __consumer_offsets topic matter for the Group Coordinator, and what can go wrong if it's misconfigured?

level: seniorimportance: should knowfreq 30%
basics
~10 s

The coordinator's state and committed offsets live in __consumer_offsets. If its replication factor is too low (e.g. 1), losing that broker loses group/offset state and breaks coordinator failover, causing duplicate processing or stuck groups.

open as a page

What is a consumer heartbeat in Kafka, and what does it accomplish?

level: juniorimportance: must knowfreq 70%
basics
~20 s

A heartbeat is a small periodic signal a consumer sends to the group coordinator broker to prove it is alive and still part of its consumer group. If heartbeats stop, the broker assumes the consumer died and reassigns its partitions.

open as a page

Explain the relationship between heartbeat.interval.ms and session.timeout.ms and how you would tune them.

level: middleimportance: must knowfreq 75%
basics
~20 s

heartbeat.interval.ms is how often the consumer sends heartbeats; session.timeout.ms is how long the coordinator waits without one before declaring the consumer dead. The interval must be well below the timeout — the rule of thumb is heartbeat.interval.ms <= session.timeout.ms / 3.

open as a page

Walk through how the coordinator detects a dead member via missed heartbeats and what happens next.

level: seniorimportance: must knowfreq 60%
basics
~20 s

The coordinator tracks each member's last heartbeat. If a member sends none for session.timeout.ms, the coordinator declares it dead, removes it from the group, and starts a rebalance that reassigns the dead member's partitions to the remaining consumers.

open as a page

What are group.min.session.timeout.ms and group.max.session.timeout.ms, and what happens if a consumer requests a session timeout outside them?

level: seniorimportance: should knowfreq 45%
basics
~20 s

They are broker-side limits on the session.timeout.ms a consumer may request: group.min.session.timeout.ms (default 6000 ms) and group.max.session.timeout.ms (default 1800000 ms). A consumer asking for a value outside this range is rejected and cannot join the group.

open as a page

Since KIP-62 split heartbeating from poll(), how do session.timeout.ms and max.poll.interval.ms differ, and why was the split necessary?

level: seniorimportance: should knowfreq 55%
basics
~20 s

session.timeout.ms detects a crashed or network-isolated consumer via missed heartbeats from the background thread. max.poll.interval.ms detects a consumer that is alive but stuck/slow in processing because it hasn't called poll() in time. Before KIP-62 both were conflated into the single session timeout.

open as a page

What is a consumer group rebalance in Kafka, and what triggers one?

level: juniorimportance: must knowfreq 78%
basics
~10 s

A rebalance is when Kafka redistributes a topic's partitions among the consumers in a group. It is triggered when a consumer joins or leaves the group, or when the set of subscribed partitions changes.

open as a page

Compare RangeAssignor, RoundRobinAssignor, and StickyAssignor. When would you pick each?

level: middleimportance: must knowfreq 70%
basics
~10 s

RangeAssignor assigns contiguous partition ranges per topic (can imbalance with multiple topics). RoundRobinAssignor spreads all partitions evenly across consumers. StickyAssignor balances evenly while trying to keep previous assignments stable across rebalances.

open as a page

Explain the difference between eager (stop-the-world) and incremental cooperative rebalancing (KIP-429).

level: seniorimportance: must knowfreq 62%
basics
~20 s

In eager rebalancing every consumer revokes all its partitions and stops processing until a new assignment arrives (stop-the-world). Incremental cooperative rebalancing (KIP-429) only revokes the partitions that actually need to move, so most partitions keep being processed throughout.

open as a page

How does a ConsumerRebalanceListener's callback behavior differ between eager and cooperative rebalancing, and where should you commit offsets?

level: seniorimportance: should knowfreq 40%
basics
~10 s

ConsumerRebalanceListener has onPartitionsRevoked, onPartitionsAssigned, and onPartitionsLost. Commit offsets in onPartitionsRevoked before giving partitions up. Under eager you're given all partitions; under cooperative you only get the ones actually moving.

open as a page

Your consumer group experiences frequent, expensive rebalances during deploys and under load. What assignor and configs would you tune, and why?

level: principalimportance: should knowfreq 38%
basics
~10 s

Switch to CooperativeStickyAssignor so most partitions keep running during rebalances, enable static membership (group.instance.id) so restarts don't trigger rebalances, and tune session.timeout.ms / heartbeat.interval.ms / max.poll.interval.ms so consumers aren't falsely evicted.

open as a page

What is a committed offset in Kafka, and what value does a consumer actually store when it commits?

level: juniorimportance: must knowfreq 80%
basics
~20 s

A committed offset records how far a consumer group has processed in a partition. It stores the offset of the NEXT message to read (last processed + 1), so a restart resumes from there without re-reading.

open as a page

Compare automatic offset commits (enable.auto.commit) with manual commits. What are the trade-offs and failure modes?

level: middleimportance: must knowfreq 85%
basics
~20 s

With enable.auto.commit=true, the consumer commits the current position periodically (auto.commit.interval.ms, default 5000ms) during poll(). It's simple but can lose or duplicate records on crash. Manual commits (commitSync/commitAsync after processing) give you control over timing and delivery semantics.

open as a page

You need at-least-once delivery with no data loss in a consumer. How do you order processing and commits, and how do you bound duplicates?

level: seniorimportance: must knowfreq 70%
basics
~20 s

Disable auto-commit, process the records first, then commit. Never commit before the work is durable. Because a crash between processing and commit replays the batch, make handlers idempotent. For exactly-once, use Kafka transactions or commit offsets atomically with your output to an external store.

open as a page

Explain the __consumer_offsets topic: why it's log-compacted, how offsets are keyed, and how the group coordinator uses it.

level: seniorimportance: should knowfreq 45%
basics
~20 s

__consumer_offsets is an internal Kafka topic (50 partitions by default) where consumer commits are stored as messages keyed by (group, topic, partition). It's log-compacted so only the latest committed offset per key is retained. The group coordinator broker reads/writes it to track group progress.

open as a page

How do you commit offsets safely around a consumer group rebalance using ConsumerRebalanceListener?

level: seniorimportance: should knowfreq 60%
basics
~20 s

Register a ConsumerRebalanceListener via subscribe(). In onPartitionsRevoked, commit offsets for partitions you're about to lose before they move to another consumer. onPartitionsAssigned runs when you gain partitions (e.g., to seek to a custom position). This prevents duplicate or lost processing across rebalances.

open as a page

What does the consumer config auto.offset.reset control, and what do its values earliest, latest, and none do?

level: juniorimportance: must knowfreq 78%
basics
~10 s

auto.offset.reset decides where a consumer starts reading when there is no valid committed offset for a partition. earliest starts at the oldest message, latest at the newest, and none throws an exception.

open as a page

Explain seek(), seekToBeginning(), and seekToEnd() — how do they work and when does the new position take effect?

level: middleimportance: must knowfreq 64%
basics
~10 s

These methods manually set where the consumer reads next. seek() jumps to a specific offset; seekToBeginning() to the oldest; seekToEnd() to the newest. The change takes effect on the next poll(), and overrides auto.offset.reset.

open as a page

What is the difference between position(), committed(), and beginningOffsets()/endOffsets()?

level: middleimportance: should knowfreq 52%
basics
~10 s

position() is where the consumer will read next (in-memory). committed() is the last offset persisted to Kafka for the group. beginningOffsets()/endOffsets() are the oldest and newest (end) offsets currently in each partition.

open as a page

How do you make a consumer start reading from a specific point in time, and what does offsetsForTimes return at the edges?

level: seniorimportance: should knowfreq 48%
basics
~20 s

Use offsetsForTimes(map of partition to timestamp) to look up the first offset whose record timestamp is >= the given time, then seek() to it. If no record has a timestamp at or after the target, it returns null for that partition.

open as a page

Design how you'd reprocess a topic from a specific point for a running consumer group without losing the current state, and discuss the tradeoffs of the available mechanisms.

level: principalimportance: should knowfreq 34%
basics
~10 s

Use the kafka-consumer-groups CLI --reset-offsets (to earliest, a timestamp, or a specific offset) while the group is stopped, or seek() the partitions in code. auto.offset.reset can't do it because committed offsets already exist.

open as a page

How do you inspect consumer lag from the command line, and what do the columns of kafka-consumer-groups.sh --describe mean?

level: juniorimportance: must knowfreq 70%
basics
~10 s

Run kafka-consumer-groups.sh --bootstrap-server host:port --describe --group <group>. It lists each partition with CURRENT-OFFSET (committed), LOG-END-OFFSET (latest), and LAG = the difference, plus the owning consumer.

open as a page

What is consumer lag in Kafka, and how is it calculated for a single partition?

level: juniorimportance: must knowfreq 80%
basics
~10 s

Consumer lag is how far behind a consumer is on a partition: the latest message offset (log-end-offset) minus the offset the consumer has committed. Lag of 0 means fully caught up.

open as a page

What is the records-lag-max JMX metric, where does it come from, and what are its strengths and blind spots compared to broker-side lag?

level: middleimportance: must knowfreq 60%
basics
~10 s

records-lag-max is a client-side JMX metric exposed by each consumer reporting the maximum lag across the partitions it currently fetches. It's cheap and real-time but only sees assigned partitions of running consumers.

open as a page

Why use a dedicated lag monitor like Burrow or Kafka Lag Exporter instead of just the CLI or records-lag-max, and how do their approaches differ?

level: seniorimportance: should knowfreq 45%
basics
~20 s

Dedicated monitors continuously read every group's committed offsets and partition LEOs externally, so they see lag even when consumers are dead, and expose it to Prometheus/alerting. Burrow adds a threshold-free status evaluation; Lag Exporter adds time-based lag estimates.

open as a page

How do you diagnose partition skew from lag data, and how would you design lag-driven autoscaling for a consumer group?

level: principalimportance: should knowfreq 35%
basics
~20 s

Skew shows as lag concentrated on a few partitions while others sit at zero — usually hot keys or a stuck/slow consumer. For autoscaling, scale consumers on total/time lag, but cap replicas at the partition count, throttle on rebalances, and prefer time-lag with hysteresis to avoid flapping.

open as a page

What is static membership in Kafka consumer groups, and which config enables it?

level: juniorimportance: must knowfreq 60%
basics
~20 s

Static membership gives a consumer a stable identity via the group.instance.id config (KIP-345). With it, a brief consumer restart does not trigger a group rebalance, because the broker recognizes the returning member by its fixed id.

open as a page

How does static membership skip a rebalance when a consumer restarts, and what is session.timeout.ms's role?

level: middleimportance: must knowfreq 55%
basics
~20 s

A static member keeps its slot during a brief restart because it doesn't send LeaveGroup and the coordinator hasn't expired its session yet. If it rejoins within session.timeout.ms with the same group.instance.id, it gets the same partitions back — no rebalance.

open as a page

What is the FencedInstanceIdException, and when does a static consumer get fenced?

level: seniorimportance: should knowfreq 40%
basics
~20 s

If two live consumers join the same group with the same group.instance.id, the broker treats them as duplicates of one static member and fences the older one, throwing FencedInstanceIdException so only one instance keeps consuming each set of partitions.

open as a page

How do you tune static membership for zero-rebalance rolling restarts, and what trade-off does the session timeout introduce?

level: seniorimportance: should knowfreq 45%
basics
~20 s

Give each instance a stable group.instance.id and set session.timeout.ms longer than a single instance's worst-case restart time. Then rolling-restart one instance at a time so each returns inside its window — no rebalances. The trade-off: a longer timeout slows detection of genuinely crashed instances.

open as a page

How does static membership differ from cooperative (incremental) rebalancing, and when would you use each or both?

level: principalimportance: should knowfreq 35%
basics
~20 s

Static membership (group.instance.id) avoids rebalances entirely for transient restarts. Cooperative-sticky rebalancing makes rebalances that DO happen incremental — only affected partitions move instead of stopping the whole group. They solve different problems and are best used together.

open as a page

What is max.poll.interval.ms, and what happens if your consumer takes longer than this value to process a batch of records?

level: juniorimportance: must knowfreq 78%
basics
~20 s

max.poll.interval.ms is the maximum time allowed between two poll() calls. If processing a batch takes longer, Kafka assumes the consumer is stuck, kicks it out of the group, and rebalances its partitions to other members.

open as a page

Explain the pause()/resume() backpressure pattern for offloading long-running record processing to worker threads while keeping the consumer in its group.

level: seniorimportance: must knowfreq 56%
basics
~20 s

Hand records to a worker pool for slow processing. To avoid fetching more than you can handle, call consumer.pause() on the assigned partitions and keep calling poll() (which now returns nothing but proves liveness). When workers catch up, call resume(). Commit only completed offsets.

open as a page

How does max.poll.interval.ms relate to session.timeout.ms and heartbeat.interval.ms? Why were poll liveness and heartbeat liveness decoupled?

level: middleimportance: should knowfreq 64%
basics
~10 s

session.timeout.ms / heartbeat.interval.ms detect a dead consumer via a background heartbeat thread. max.poll.interval.ms detects a live-but-stuck consumer via the poll() call. They were split (KIP-62) so slow processing doesn't get confused with a crash.

open as a page

A consumer is being evicted because each batch takes too long to process. How does reducing max.poll.records help, and what are the tradeoffs?

level: middleimportance: should knowfreq 58%
basics
~20 s

max.poll.records caps how many records each poll() returns. Lowering it means smaller batches that process faster, so poll() is called again sooner and you stay under max.poll.interval.ms. The tradeoff is more poll round-trips and potentially lower throughput.

open as a page

As a principal engineer, how would you decide between raising max.poll.interval.ms, lowering max.poll.records, and offloading processing for a consumer with slow, variable per-record processing? What are the systemic risks of just cranking the interval up?

level: principalimportance: should knowfreq 40%
basics
~20 s

Match the lever to the cost model: lower max.poll.records if cost scales with count; offload with pause()/resume() if individual records are slow or variable; raise max.poll.interval.ms only for legitimately long, bounded work. Cranking the interval high delays detection of truly stuck consumers, slowing partition reassignment and recovery.

open as a page

What is the difference between subscribe() and assign() on a Kafka consumer, and when would you choose each?

level: juniorimportance: must knowfreq 75%
basics
~10 s

subscribe() joins a consumer group and lets Kafka assign partitions automatically (dynamic, rebalances on membership change). assign() manually pins specific partitions to the consumer with no group coordination or rebalancing.

open as a page

Explain the ConsumerRebalanceListener callbacks (onPartitionsRevoked, onPartitionsAssigned, onPartitionsLost) and what you should do in each.

level: seniorimportance: must knowfreq 60%
basics
~20 s

It's a callback you pass to subscribe() so you can react to rebalances: onPartitionsRevoked (commit offsets / flush state before losing partitions), onPartitionsAssigned (init state / seek for new partitions), onPartitionsLost (partitions gone unexpectedly — don't commit).

open as a page

With a consumer using assign() instead of subscribe(), how do you control where consumption starts, since there's no rebalance to trigger position setup?

level: middleimportance: should knowfreq 35%
basics
~10 s

After assign(), the consumer starts from the last committed offset for that group.id if one exists; otherwise auto.offset.reset (earliest/latest) applies. You can override explicitly with seek(), seekToBeginning(), or seekToEnd() before polling.

open as a page

How does pattern subscription (subscribe with a regex) work in Kafka, and what are its operational caveats?

level: middleimportance: should knowfreq 45%
basics
~10 s

subscribe(Pattern) matches topic names against a regex. The consumer auto-discovers new matching topics and rebalances to include them. It still uses group management, just with a dynamic topic set.

open as a page

You need deterministic, static partition ownership (each service instance always owns the same partitions) without rebalance pauses. How do you design this, and what do you give up?

level: principalimportance: should knowfreq 30%
basics
~20 s

Use assign() to pin a fixed partition set to each instance based on a stable ordinal (e.g. pod index). You avoid rebalances and get deterministic ownership, but you give up automatic load balancing and failover — you must handle scaling and dead-instance takeover yourself.

open as a page

What are key.deserializer and value.deserializer in a Kafka consumer, and why must they pair with the producer's serializers?

level: juniorimportance: must knowfreq 80%
basics
~10 s

Kafka stores message keys and values as raw bytes. key.deserializer and value.deserializer are classes that turn those bytes back into objects. They must match the producer's serializers, or the bytes won't decode correctly.

open as a page

Explain client.id, group.id, and isolation.level on a Kafka consumer — what each controls and their valid values.

level: middleimportance: must knowfreq 65%
basics
~10 s

group.id names the consumer group that shares partitions and offsets. client.id is a logical label for the connection used in logs/metrics/quotas. isolation.level (read_uncommitted or read_committed) decides whether the consumer sees records from uncommitted transactions.

open as a page

What is a poison-pill record, and how does ErrorHandlingDeserializer solve it?

level: seniorimportance: must knowfreq 70%
basics
~20 s

A poison pill is a record whose bytes can't be deserialized, so it throws on every poll and blocks the consumer forever. ErrorHandlingDeserializer wraps the real deserializer, catches the failure, and hands the error to a recoverer instead of crashing.

open as a page

When using the Confluent schema-registry Avro deserializer, what is the difference between SpecificAvro and GenericAvro deserialization, and how does specific.avro.reader control it?

level: seniorimportance: should knowfreq 50%
basics
~10 s

KafkaAvroDeserializer reads a schema-registry ID from each record and fetches the writer schema. With specific.avro.reader=false (default) you get a generic GenericRecord; with true you get generated, typed SpecificRecord classes.

open as a page

What is the security risk of deserializing untrusted Kafka payloads, and how do you mitigate it (e.g., with JsonDeserializer trusted packages)?

level: principalimportance: should knowfreq 40%
basics
~20 s

If a deserializer instantiates arbitrary Java classes named in the payload (e.g., via type headers or Java native serialization), an attacker who controls the bytes can trigger remote code execution. Mitigate by restricting allowed types — e.g., Spring's JsonDeserializer trusted.packages allow-list.

open as a page

When a Kafka consumer switches to group.protocol=consumer, which settings move to the broker and which client configs stop working?

level: middleimportance: must knowfreq 48%
basics
~10 s

Under group.protocol=consumer the broker owns assignment, heartbeat interval and session timeout. Setting partition.assignment.strategy, heartbeat.interval.ms or session.timeout.ms on the client throws a ConfigException at construction, and enforceRebalance() only logs a warning.

open as a page

What problem does Kafka 4.0's new consumer group protocol (KIP-848, group.protocol=consumer) solve compared with the classic rebalance protocol?

level: juniorimportance: should knowfreq 55%
basics
~20 s

It removes the classic protocol's group-wide rebalance barrier and its client-side leader: the broker's group coordinator computes assignments, and each consumer reconciles incrementally through heartbeats, so one member's change no longer stalls the whole group.

open as a page

Under Kafka's KIP-848 consumer group protocol, how do the group, target-assignment and member epochs drive reconciliation, and when is a member fenced?

level: seniorimportance: should knowfreq 30%
basics
~20 s

The coordinator bumps the group epoch on membership, subscription or metadata changes and computes a target at that epoch; each member revokes partitions moving away, then advances its member epoch and takes freed ones. Stale epochs or missed deadlines fence it.

open as a page

A payments team on a 3.9 cluster upgraded to Kafka 4.x rolls 60 consumers to group.protocol=consumer one pod at a time; how does the online conversion work, and what can block it?

level: seniorimportance: should knowfreq 35%
basics
~20 s

The first consumer-protocol pod's heartbeat converts the classic group; the remaining classic pods are served through JoinGroup and SyncGroup. Conversion is refused if the migration policy forbids it, the group exceeds group.consumer.max.size, or a custom assignor embeds metadata.

open as a page

As a platform lead, would you move your Kafka consumer fleet to group.protocol=consumer now or wait for the 5.0 default flip, and how would you stage it?

level: principalimportance: should knowfreq 22%
basics
~20 s

Usually migrate deliberately before 5.0 rather than inherit the flip: inventory blockers such as custom assignors and enforceRebalance calls, settle who owns broker-side timeouts, then roll group by group behind the migration policy guardrail with rollback rehearsed.

open as a page

What is a Kafka share group, introduced by KIP-932, and how does it differ from a Kafka consumer group?

level: juniorimportance: should knowfreq 45%
basics
~20 s

A share group is a Kafka group type whose members share partitions: the broker locks each record for one member, which acknowledges it individually, so the group can outgrow the partition count at the cost of ordering.

open as a page

In a KafkaShareConsumer, how do implicit and explicit acknowledgement differ, and how does share.acknowledgement.mode change the poll loop?

level: middleimportance: should knowfreq 30%
basics
~10 s

With share.acknowledgement.mode=implicit, the default, the next poll() or commit accepts the whole previous batch; with explicit, the application calls acknowledge(record, type) for every record before polling again, which allows per-record RELEASE, REJECT and RENEW.

open as a page

In a Kafka share group, how do acquisition locks and the AcknowledgeType values ACCEPT, RELEASE, REJECT and RENEW decide a record's fate?

level: middleimportance: should knowfreq 38%
basics
~20 s

Fetching acquires each record for one share consumer under a lock (30 s by default); ACCEPT finishes it, RELEASE returns it for another attempt, REJECT archives it, RENEW extends the lock, and an expired lock returns it automatically.

open as a page

A Kafka share group runs 40 thumbnail workers on a 6-partition topic, and one corrupt image keeps crashing workers; how do delivery counting and locks contain it, and what would you change?

level: seniorimportance: should knowfreq 28%
basics
~20 s

Each crash leaves the image locked until the lock expires or the broker frees it, then it is redelivered; at the delivery count limit (5 by default) it is archived. Catching the error and acknowledging REJECT archives it at once.

open as a page

When should a team run a Kafka workload on a share group instead of a consumer group or a separate queue broker, and what does adopting share groups cost?

level: principalimportance: nice to knowfreq 24%
basics
~20 s

Use a share group for independent jobs that need more workers than partitions or per-record retry; keep a consumer group for ordering, keyed state or exactly-once; and budget a 4.2+ cluster, share.version, a client rewrite and new tooling.

open as a page