skip to content

What is a consumer heartbeat in Kafka, and what does it accomplish?

level: juniorimportance: must knowfreq 70%

answer

  1. background thread, not poll()
  2. tells coordinator 'I'm alive'
  3. miss it for session.timeout.ms -> dead
  4. death triggers rebalance
  5. KIP-62 split heartbeat from poll

basics

~20 s

A heartbeat is a small periodic signal a consumer sends to the group coordinator broker to prove it is alive and still part of its consumer group. If heartbeats stop, the broker assumes the consumer died and reassigns its partitions.

solid answer

~40 s

A heartbeat is a lightweight message a Kafka consumer periodically sends to the group coordinator (the broker that manages its consumer group) to signal liveness and confirm continued membership. In modern clients a dedicated background heartbeat thread sends them on the interval set by heartbeat.interval.ms, independent of the application's poll() calls. The coordinator tracks the last heartbeat per member; if none arrives within session.timeout.ms, it marks the member dead, removes it from the group, and triggers a rebalance so the dead member's partitions get reassigned to surviving members. Heartbeats are how the group stays aware of which members are present, which underpins partition assignment and failure detection.

go deeper

for a junior

Know it's a periodic 'I'm alive' signal to the broker; missing it gets you kicked out of the group.

for a middle

Know the background-thread model, that it goes to the group coordinator, and that timeout triggers a rebalance.

for a senior

Distinguish heartbeat liveness from processing progress (max.poll.interval.ms), and explain LeaveGroup on graceful shutdown.

for a principal

Reason about failure-detection latency vs false-positive evictions and how heartbeat design choices affect group stability at scale.

## What problem heartbeats solve A Kafka **consumer group** is a set of consumer instances that cooperatively read from a topic's partitions, with each partition assigned to exactly one member. To keep that assignment correct, the cluster must know which members are alive. The **group coordinator** — a specific broker chosen to manage a given group — needs a continuous signal that each member is healthy. That signal is the **heartbeat**. ## Mechanism A heartbeat is a small RPC (a `Heartbeat` request) the consumer sends to the coordinator. In modern Java clients (since KIP-62, Kafka 0.10.1) heartbeats are sent by a **dedicated background thread** in the consumer, not on the application thread. This thread fires roughly every `heartbeat.interval.ms` (default 3000 ms). The coordinator records the timestamp of each member's last heartbeat. ## What it accomplishes - **Liveness / failure detection:** if the coordinator does not hear from a member within `session.timeout.ms` (default 45000 ms in recent versions), it declares the member dead, evicts it from the group, and starts a **rebalance** to reassign that member's partitions to the survivors. - **Rebalance signaling:** the heartbeat response also carries a flag telling the consumer when a rebalance is in progress, so the consumer knows it must rejoin the group. - **Membership awareness:** heartbeats are the heartbeat (pun intended) of the membership protocol that drives partition assignment. ## Edge cases / nuances - Heartbeats prove the *network/thread* is alive, not that the application is making progress. A consumer can heartbeat fine while being stuck in slow processing; that case is covered by a separate timer, `max.poll.interval.ms`, not by the heartbeat path. - If the background thread dies or the consumer is GC-paused longer than the session timeout, the member is evicted even though the process technically exists. - A graceful shutdown sends a `LeaveGroup` request so the coordinator can rebalance immediately rather than waiting for the session to expire.

  • Which broker receives the heartbeats?
    The group coordinator — the specific broker elected to manage that consumer group (chosen by hashing the group.id onto a __consumer_offsets partition).
  • Does a heartbeat prove the application is processing records?
    No. It only proves the consumer's background thread and network are alive. Stuck application processing is detected by max.poll.interval.ms, a separate mechanism.

saying these in an interview costs you the question

  • Saying heartbeats are sent inside poll() in modern clients (KIP-62 moved them to a background thread).
  • Claiming a healthy heartbeat means the app is making processing progress.
  • Confusing the heartbeat with the produce/fetch data path.

context

resolving links…