Consumers and Consumer Groups
The read path: the poll loop, consumer groups and rebalancing, offset commits, and lag. Interviewers spend most of their Kafka time here because nearly every production issue surfaces as a consumer symptom.
part ofApache Kafkaread it →on this pageshowhide
explore
- Poll Loop and Fetch Mechanics5 questions
- Heartbeat and Session Liveness5 questions
- Rebalancing and Assignment Strategies5 questions
- Offset Management and Commit Strategies5 questions
- Position, Seek and Offset Reset5 questions
- Consumer Lag and Measurement5 questions
- Static Membership5 questions
- Deserialization and Consumer Configuration5 questions
- KIP-848 Rebalance Protocol5 questions
- KIP-932 Queues5 questions
questions
65 · 13 sectionsWhat does KafkaConsumer.poll(Duration) actually do, and why must you call it in a continuous loop?
basics
~20 spoll(Duration) fetches a batch of records the consumer already has buffered (or waits up to the timeout for some), and also drives heartbeats, group coordination, and offset commits. You loop because all consumer work happens during poll.
What does max.poll.records control, and how does it differ from the fetch-size settings?
basics
~20 smax.poll.records caps how many records a single poll() call returns to your code (default 500). Fetch-size settings (fetch.max.bytes, max.partition.fetch.bytes) control how much data is pulled from brokers over the wire — bytes, not record count.
Explain how fetch.min.bytes and fetch.max.wait.ms work together to trade off latency vs throughput in consumer fetches.
basics
~20 sfetch.min.bytes tells the broker the minimum data to accumulate before answering a fetch; the broker waits up to fetch.max.wait.ms for that much to arrive, then responds anyway. Bigger min.bytes = better throughput but higher latency.
What is max.partition.fetch.bytes, how does it relate to fetch.max.bytes, and what failure mode arises if a record exceeds it?
basics
~20 smax.partition.fetch.bytes caps bytes returned per partition per fetch (default ~1MB); fetch.max.bytes caps total bytes across all partitions (~50MB). Both are soft limits: to avoid stalling, a broker always returns at least one full record batch even if it exceeds the cap.
Describe how the consumer's per-partition fetch buffers and prefetch pipelining work to keep poll() fast and overlap network with processing.
basics
~20 sThe consumer keeps a buffer of records per partition and issues fetch requests for more data BEFORE the buffer is empty (prefetch). So while you process the current batch, the next fetch is already in flight, and poll() usually returns instantly from the buffer instead of waiting on the network.
Consumer Group Protocol and Group Coordinator
all 5 Consumer Group Protocol and Group Coordinator questions →What is the Group Coordinator in Kafka, and how is the coordinator broker for a particular consumer group chosen?
basics
~10 sThe Group Coordinator is a broker that manages a consumer group: it handles members joining/leaving, triggers rebalances, and stores offsets. The coordinator is the broker that leads the __consumer_offsets partition the group maps to.
Walk through the JoinGroup and SyncGroup phases of a rebalance. Who computes the partition assignment, and how does it reach each member?
basics
~20 sAll members send JoinGroup; the coordinator picks one member as group leader and returns the full member list to it. The leader computes assignments and sends them in SyncGroup; the coordinator fans them out to each member in their SyncGroup response.
Describe the consumer group states the coordinator maintains (Empty, PreparingRebalance, CompletingRebalance, Stable, Dead) and the transitions between them.
basics
~10 sThe coordinator tracks a group's lifecycle: Empty (no members), PreparingRebalance (waiting for members to join), CompletingRebalance (awaiting SyncGroup), Stable (assignment done, consuming), and Dead (group removed). Membership changes drive transitions through these states.
What are the generation ID and member.id in the consumer group protocol, and how do they protect against stale/zombie members?
basics
~20 sThe generation ID is a monotonically increasing counter the coordinator bumps on each successful rebalance; member.id uniquely identifies a member within the group. Requests carrying a stale generation or unknown member.id are rejected, fencing zombies.
Why does the replication factor and configuration of the __consumer_offsets topic matter for the Group Coordinator, and what can go wrong if it's misconfigured?
basics
~10 sThe coordinator's state and committed offsets live in __consumer_offsets. If its replication factor is too low (e.g. 1), losing that broker loses group/offset state and breaks coordinator failover, causing duplicate processing or stuck groups.
What is a consumer heartbeat in Kafka, and what does it accomplish?
basics
~20 sA heartbeat is a small periodic signal a consumer sends to the group coordinator broker to prove it is alive and still part of its consumer group. If heartbeats stop, the broker assumes the consumer died and reassigns its partitions.
Explain the relationship between heartbeat.interval.ms and session.timeout.ms and how you would tune them.
basics
~20 sheartbeat.interval.ms is how often the consumer sends heartbeats; session.timeout.ms is how long the coordinator waits without one before declaring the consumer dead. The interval must be well below the timeout — the rule of thumb is heartbeat.interval.ms <= session.timeout.ms / 3.
Walk through how the coordinator detects a dead member via missed heartbeats and what happens next.
basics
~20 sThe coordinator tracks each member's last heartbeat. If a member sends none for session.timeout.ms, the coordinator declares it dead, removes it from the group, and starts a rebalance that reassigns the dead member's partitions to the remaining consumers.
What are group.min.session.timeout.ms and group.max.session.timeout.ms, and what happens if a consumer requests a session timeout outside them?
basics
~20 sThey are broker-side limits on the session.timeout.ms a consumer may request: group.min.session.timeout.ms (default 6000 ms) and group.max.session.timeout.ms (default 1800000 ms). A consumer asking for a value outside this range is rejected and cannot join the group.
Since KIP-62 split heartbeating from poll(), how do session.timeout.ms and max.poll.interval.ms differ, and why was the split necessary?
basics
~20 ssession.timeout.ms detects a crashed or network-isolated consumer via missed heartbeats from the background thread. max.poll.interval.ms detects a consumer that is alive but stuck/slow in processing because it hasn't called poll() in time. Before KIP-62 both were conflated into the single session timeout.
What is a consumer group rebalance in Kafka, and what triggers one?
basics
~10 sA rebalance is when Kafka redistributes a topic's partitions among the consumers in a group. It is triggered when a consumer joins or leaves the group, or when the set of subscribed partitions changes.
Compare RangeAssignor, RoundRobinAssignor, and StickyAssignor. When would you pick each?
basics
~10 sRangeAssignor assigns contiguous partition ranges per topic (can imbalance with multiple topics). RoundRobinAssignor spreads all partitions evenly across consumers. StickyAssignor balances evenly while trying to keep previous assignments stable across rebalances.
Explain the difference between eager (stop-the-world) and incremental cooperative rebalancing (KIP-429).
basics
~20 sIn eager rebalancing every consumer revokes all its partitions and stops processing until a new assignment arrives (stop-the-world). Incremental cooperative rebalancing (KIP-429) only revokes the partitions that actually need to move, so most partitions keep being processed throughout.
How does a ConsumerRebalanceListener's callback behavior differ between eager and cooperative rebalancing, and where should you commit offsets?
basics
~10 sConsumerRebalanceListener has onPartitionsRevoked, onPartitionsAssigned, and onPartitionsLost. Commit offsets in onPartitionsRevoked before giving partitions up. Under eager you're given all partitions; under cooperative you only get the ones actually moving.
Your consumer group experiences frequent, expensive rebalances during deploys and under load. What assignor and configs would you tune, and why?
basics
~10 sSwitch to CooperativeStickyAssignor so most partitions keep running during rebalances, enable static membership (group.instance.id) so restarts don't trigger rebalances, and tune session.timeout.ms / heartbeat.interval.ms / max.poll.interval.ms so consumers aren't falsely evicted.
What is a committed offset in Kafka, and what value does a consumer actually store when it commits?
basics
~20 sA committed offset records how far a consumer group has processed in a partition. It stores the offset of the NEXT message to read (last processed + 1), so a restart resumes from there without re-reading.
Compare automatic offset commits (enable.auto.commit) with manual commits. What are the trade-offs and failure modes?
basics
~20 sWith enable.auto.commit=true, the consumer commits the current position periodically (auto.commit.interval.ms, default 5000ms) during poll(). It's simple but can lose or duplicate records on crash. Manual commits (commitSync/commitAsync after processing) give you control over timing and delivery semantics.
You need at-least-once delivery with no data loss in a consumer. How do you order processing and commits, and how do you bound duplicates?
basics
~20 sDisable auto-commit, process the records first, then commit. Never commit before the work is durable. Because a crash between processing and commit replays the batch, make handlers idempotent. For exactly-once, use Kafka transactions or commit offsets atomically with your output to an external store.
Explain the __consumer_offsets topic: why it's log-compacted, how offsets are keyed, and how the group coordinator uses it.
basics
~20 s__consumer_offsets is an internal Kafka topic (50 partitions by default) where consumer commits are stored as messages keyed by (group, topic, partition). It's log-compacted so only the latest committed offset per key is retained. The group coordinator broker reads/writes it to track group progress.
How do you commit offsets safely around a consumer group rebalance using ConsumerRebalanceListener?
basics
~20 sRegister a ConsumerRebalanceListener via subscribe(). In onPartitionsRevoked, commit offsets for partitions you're about to lose before they move to another consumer. onPartitionsAssigned runs when you gain partitions (e.g., to seek to a custom position). This prevents duplicate or lost processing across rebalances.
What does the consumer config auto.offset.reset control, and what do its values earliest, latest, and none do?
basics
~10 sauto.offset.reset decides where a consumer starts reading when there is no valid committed offset for a partition. earliest starts at the oldest message, latest at the newest, and none throws an exception.
Explain seek(), seekToBeginning(), and seekToEnd() — how do they work and when does the new position take effect?
basics
~10 sThese methods manually set where the consumer reads next. seek() jumps to a specific offset; seekToBeginning() to the oldest; seekToEnd() to the newest. The change takes effect on the next poll(), and overrides auto.offset.reset.
What is the difference between position(), committed(), and beginningOffsets()/endOffsets()?
basics
~10 sposition() is where the consumer will read next (in-memory). committed() is the last offset persisted to Kafka for the group. beginningOffsets()/endOffsets() are the oldest and newest (end) offsets currently in each partition.
How do you make a consumer start reading from a specific point in time, and what does offsetsForTimes return at the edges?
basics
~20 sUse offsetsForTimes(map of partition to timestamp) to look up the first offset whose record timestamp is >= the given time, then seek() to it. If no record has a timestamp at or after the target, it returns null for that partition.
Design how you'd reprocess a topic from a specific point for a running consumer group without losing the current state, and discuss the tradeoffs of the available mechanisms.
basics
~10 sUse the kafka-consumer-groups CLI --reset-offsets (to earliest, a timestamp, or a specific offset) while the group is stopped, or seek() the partitions in code. auto.offset.reset can't do it because committed offsets already exist.
How do you inspect consumer lag from the command line, and what do the columns of kafka-consumer-groups.sh --describe mean?
basics
~10 sRun kafka-consumer-groups.sh --bootstrap-server host:port --describe --group <group>. It lists each partition with CURRENT-OFFSET (committed), LOG-END-OFFSET (latest), and LAG = the difference, plus the owning consumer.
What is consumer lag in Kafka, and how is it calculated for a single partition?
basics
~10 sConsumer lag is how far behind a consumer is on a partition: the latest message offset (log-end-offset) minus the offset the consumer has committed. Lag of 0 means fully caught up.
What is the records-lag-max JMX metric, where does it come from, and what are its strengths and blind spots compared to broker-side lag?
basics
~10 srecords-lag-max is a client-side JMX metric exposed by each consumer reporting the maximum lag across the partitions it currently fetches. It's cheap and real-time but only sees assigned partitions of running consumers.
Why use a dedicated lag monitor like Burrow or Kafka Lag Exporter instead of just the CLI or records-lag-max, and how do their approaches differ?
basics
~20 sDedicated monitors continuously read every group's committed offsets and partition LEOs externally, so they see lag even when consumers are dead, and expose it to Prometheus/alerting. Burrow adds a threshold-free status evaluation; Lag Exporter adds time-based lag estimates.
How do you diagnose partition skew from lag data, and how would you design lag-driven autoscaling for a consumer group?
basics
~20 sSkew shows as lag concentrated on a few partitions while others sit at zero — usually hot keys or a stuck/slow consumer. For autoscaling, scale consumers on total/time lag, but cap replicas at the partition count, throttle on rebalances, and prefer time-lag with hysteresis to avoid flapping.
What is static membership in Kafka consumer groups, and which config enables it?
basics
~20 sStatic membership gives a consumer a stable identity via the group.instance.id config (KIP-345). With it, a brief consumer restart does not trigger a group rebalance, because the broker recognizes the returning member by its fixed id.
How does static membership skip a rebalance when a consumer restarts, and what is session.timeout.ms's role?
basics
~20 sA static member keeps its slot during a brief restart because it doesn't send LeaveGroup and the coordinator hasn't expired its session yet. If it rejoins within session.timeout.ms with the same group.instance.id, it gets the same partitions back — no rebalance.
What is the FencedInstanceIdException, and when does a static consumer get fenced?
basics
~20 sIf two live consumers join the same group with the same group.instance.id, the broker treats them as duplicates of one static member and fences the older one, throwing FencedInstanceIdException so only one instance keeps consuming each set of partitions.
How do you tune static membership for zero-rebalance rolling restarts, and what trade-off does the session timeout introduce?
basics
~20 sGive each instance a stable group.instance.id and set session.timeout.ms longer than a single instance's worst-case restart time. Then rolling-restart one instance at a time so each returns inside its window — no rebalances. The trade-off: a longer timeout slows detection of genuinely crashed instances.
How does static membership differ from cooperative (incremental) rebalancing, and when would you use each or both?
basics
~20 sStatic membership (group.instance.id) avoids rebalances entirely for transient restarts. Cooperative-sticky rebalancing makes rebalances that DO happen incremental — only affected partitions move instead of stopping the whole group. They solve different problems and are best used together.
Max Poll Interval and Long-Processing Pitfalls
all 5 Max Poll Interval and Long-Processing Pitfalls questions →What is max.poll.interval.ms, and what happens if your consumer takes longer than this value to process a batch of records?
basics
~20 smax.poll.interval.ms is the maximum time allowed between two poll() calls. If processing a batch takes longer, Kafka assumes the consumer is stuck, kicks it out of the group, and rebalances its partitions to other members.
Explain the pause()/resume() backpressure pattern for offloading long-running record processing to worker threads while keeping the consumer in its group.
basics
~20 sHand records to a worker pool for slow processing. To avoid fetching more than you can handle, call consumer.pause() on the assigned partitions and keep calling poll() (which now returns nothing but proves liveness). When workers catch up, call resume(). Commit only completed offsets.
How does max.poll.interval.ms relate to session.timeout.ms and heartbeat.interval.ms? Why were poll liveness and heartbeat liveness decoupled?
basics
~10 ssession.timeout.ms / heartbeat.interval.ms detect a dead consumer via a background heartbeat thread. max.poll.interval.ms detects a live-but-stuck consumer via the poll() call. They were split (KIP-62) so slow processing doesn't get confused with a crash.
A consumer is being evicted because each batch takes too long to process. How does reducing max.poll.records help, and what are the tradeoffs?
basics
~20 smax.poll.records caps how many records each poll() returns. Lowering it means smaller batches that process faster, so poll() is called again sooner and you stay under max.poll.interval.ms. The tradeoff is more poll round-trips and potentially lower throughput.
As a principal engineer, how would you decide between raising max.poll.interval.ms, lowering max.poll.records, and offloading processing for a consumer with slow, variable per-record processing? What are the systemic risks of just cranking the interval up?
basics
~20 sMatch the lever to the cost model: lower max.poll.records if cost scales with count; offload with pause()/resume() if individual records are slow or variable; raise max.poll.interval.ms only for legitimately long, bounded work. Cranking the interval high delays detection of truly stuck consumers, slowing partition reassignment and recovery.
Manual Partition Assignment and Subscription Modes
all 5 Manual Partition Assignment and Subscription Modes questions →What is the difference between subscribe() and assign() on a Kafka consumer, and when would you choose each?
basics
~10 ssubscribe() joins a consumer group and lets Kafka assign partitions automatically (dynamic, rebalances on membership change). assign() manually pins specific partitions to the consumer with no group coordination or rebalancing.
Explain the ConsumerRebalanceListener callbacks (onPartitionsRevoked, onPartitionsAssigned, onPartitionsLost) and what you should do in each.
basics
~20 sIt's a callback you pass to subscribe() so you can react to rebalances: onPartitionsRevoked (commit offsets / flush state before losing partitions), onPartitionsAssigned (init state / seek for new partitions), onPartitionsLost (partitions gone unexpectedly — don't commit).
With a consumer using assign() instead of subscribe(), how do you control where consumption starts, since there's no rebalance to trigger position setup?
basics
~10 sAfter assign(), the consumer starts from the last committed offset for that group.id if one exists; otherwise auto.offset.reset (earliest/latest) applies. You can override explicitly with seek(), seekToBeginning(), or seekToEnd() before polling.
How does pattern subscription (subscribe with a regex) work in Kafka, and what are its operational caveats?
basics
~10 ssubscribe(Pattern) matches topic names against a regex. The consumer auto-discovers new matching topics and rebalances to include them. It still uses group management, just with a dynamic topic set.
You need deterministic, static partition ownership (each service instance always owns the same partitions) without rebalance pauses. How do you design this, and what do you give up?
basics
~20 sUse assign() to pin a fixed partition set to each instance based on a stable ordinal (e.g. pod index). You avoid rebalances and get deterministic ownership, but you give up automatic load balancing and failover — you must handle scaling and dead-instance takeover yourself.
Deserialization and Consumer Configuration
all 5 Deserialization and Consumer Configuration questions →What are key.deserializer and value.deserializer in a Kafka consumer, and why must they pair with the producer's serializers?
basics
~10 sKafka stores message keys and values as raw bytes. key.deserializer and value.deserializer are classes that turn those bytes back into objects. They must match the producer's serializers, or the bytes won't decode correctly.
Explain client.id, group.id, and isolation.level on a Kafka consumer — what each controls and their valid values.
basics
~10 sgroup.id names the consumer group that shares partitions and offsets. client.id is a logical label for the connection used in logs/metrics/quotas. isolation.level (read_uncommitted or read_committed) decides whether the consumer sees records from uncommitted transactions.
What is a poison-pill record, and how does ErrorHandlingDeserializer solve it?
basics
~20 sA poison pill is a record whose bytes can't be deserialized, so it throws on every poll and blocks the consumer forever. ErrorHandlingDeserializer wraps the real deserializer, catches the failure, and hands the error to a recoverer instead of crashing.
When using the Confluent schema-registry Avro deserializer, what is the difference between SpecificAvro and GenericAvro deserialization, and how does specific.avro.reader control it?
basics
~10 sKafkaAvroDeserializer reads a schema-registry ID from each record and fetches the writer schema. With specific.avro.reader=false (default) you get a generic GenericRecord; with true you get generated, typed SpecificRecord classes.
What is the security risk of deserializing untrusted Kafka payloads, and how do you mitigate it (e.g., with JsonDeserializer trusted packages)?
basics
~20 sIf a deserializer instantiates arbitrary Java classes named in the payload (e.g., via type headers or Java native serialization), an attacker who controls the bytes can trigger remote code execution. Mitigate by restricting allowed types — e.g., Spring's JsonDeserializer trusted.packages allow-list.
When a Kafka consumer switches to group.protocol=consumer, which settings move to the broker and which client configs stop working?
basics
~10 sUnder group.protocol=consumer the broker owns assignment, heartbeat interval and session timeout. Setting partition.assignment.strategy, heartbeat.interval.ms or session.timeout.ms on the client throws a ConfigException at construction, and enforceRebalance() only logs a warning.
What problem does Kafka 4.0's new consumer group protocol (KIP-848, group.protocol=consumer) solve compared with the classic rebalance protocol?
basics
~20 sIt removes the classic protocol's group-wide rebalance barrier and its client-side leader: the broker's group coordinator computes assignments, and each consumer reconciles incrementally through heartbeats, so one member's change no longer stalls the whole group.
Under Kafka's KIP-848 consumer group protocol, how do the group, target-assignment and member epochs drive reconciliation, and when is a member fenced?
basics
~20 sThe coordinator bumps the group epoch on membership, subscription or metadata changes and computes a target at that epoch; each member revokes partitions moving away, then advances its member epoch and takes freed ones. Stale epochs or missed deadlines fence it.
A payments team on a 3.9 cluster upgraded to Kafka 4.x rolls 60 consumers to group.protocol=consumer one pod at a time; how does the online conversion work, and what can block it?
basics
~20 sThe first consumer-protocol pod's heartbeat converts the classic group; the remaining classic pods are served through JoinGroup and SyncGroup. Conversion is refused if the migration policy forbids it, the group exceeds group.consumer.max.size, or a custom assignor embeds metadata.
As a platform lead, would you move your Kafka consumer fleet to group.protocol=consumer now or wait for the 5.0 default flip, and how would you stage it?
basics
~20 sUsually migrate deliberately before 5.0 rather than inherit the flip: inventory blockers such as custom assignors and enforceRebalance calls, settle who owns broker-side timeouts, then roll group by group behind the migration policy guardrail with rollback rehearsed.