skip to content

Manual Partition Assignment and Subscription Modes

3 roadmaps5 questions

Choosing between subscribe() with group management and assign() with partitions you pin yourself, plus the rebalance listener callbacks. Interviewers ask when they want to see you reason about when group coordination is not wanted.

part ofApache Kafkaoverview, primer and where to startread it →
on this pageshow

explore

questions

5

What is the difference between subscribe() and assign() on a Kafka consumer, and when would you choose each?

level: juniorimportance: must knowfreq 75%

answer

  1. subscribe = group + auto rebalance
  2. assign = manual, no group, no failover
  3. IllegalStateException if you mix them
  4. assign() can still commit offsets with group.id
  5. TopicPartition list for assign()

basics

~10 s

subscribe() joins a consumer group and lets Kafka assign partitions automatically (dynamic, rebalances on membership change). assign() manually pins specific partitions to the consumer with no group coordination or rebalancing.

solid answer

~40 s

subscribe(topics) opts the consumer into group management: it joins a consumer group, the group coordinator + a leader assign partitions across members, and partitions rebalance automatically when members join/leave. You get automatic fault tolerance and load balancing but no control over which partitions you own. assign(partitions) is self-managed: you hand the consumer an explicit list of TopicPartitions, it consumes exactly those with no group membership, no rebalances, and no automatic failover. Use subscribe() for elastic, fault-tolerant consumer groups (the common case). Use assign() when you need deterministic, static partition ownership — e.g. pinning partition N to instance N for stable parallelism, replaying specific partitions, or building your own assignment logic. You cannot mix the two on one consumer instance: calling both throws IllegalStateException.

go deeper

for a junior

Know the one-line difference: subscribe = automatic/dynamic, assign = manual/static.

for a middle

Explain group coordinator + rebalances for subscribe, and no failover for assign; know they're mutually exclusive.

for a senior

Articulate concrete use cases for assign() (static parallelism, replay) and the offset-commit nuance.

for a principal

Reason about when to abandon group management entirely for deterministic ownership and how to coordinate failover externally.

## The two subscription modes A Kafka topic is split into **partitions** — ordered, independently-consumable logs. A **consumer** reads records from one or more partitions. There are two completely different ways to decide *which* partitions a given consumer instance reads. ### subscribe() — group-managed (dynamic) ```java consumer.subscribe(List.of("orders")); ``` This enrolls the consumer in a **consumer group** (identified by `group.id`). Kafka then does the work for you: - The consumer finds the **group coordinator** (a broker) and sends a JoinGroup request. - One member becomes the **group leader** and computes a partition→member assignment using the configured `partition.assignment.strategy` (e.g. CooperativeStickyAssignor). - The assignment is distributed via SyncGroup. Each member is told which partitions it owns. - When a member joins, leaves, or dies (session timeout), a **rebalance** runs and partitions are redistributed. Benefits: automatic load balancing and failover. Cost: you don't control which partitions land where, and rebalances cause brief pauses. ### assign() — self-managed (static / manual) ```java consumer.assign(List.of(new TopicPartition("orders", 3))); ``` This bypasses group management entirely. You pass an explicit list of `TopicPartition` objects and the consumer reads exactly those. There is: - **No group membership** (no JoinGroup, no coordinator-driven assignment). - **No rebalancing** — the set never changes unless *you* call assign() again. - **No automatic failover** — if this instance dies, nobody takes over its partitions. Note: even with assign(), a `group.id` can still be set so the consumer can **commit offsets** to the `__consumer_offsets` topic — assign() opts out of *partition assignment*, not necessarily offset storage. ### Mutual exclusivity A single consumer instance is in exactly one mode for its lifetime of a call sequence. Calling `assign()` after `subscribe()` (or vice versa) without first calling `unsubscribe()`/clearing throws **`IllegalStateException`** ("Subscription to topics, partitions and pattern are mutually exclusive"). ### When to pick which - **subscribe():** the default. Elastic worker pools, scaling consumers up/down, needing fault tolerance. - **assign():** deterministic ownership — e.g. a stateful stream processor that pins partition i to pod i, replay/debug of one partition, or custom external coordination (e.g. via Kubernetes StatefulSet ordinals).

  • Can a consumer using assign() still commit offsets to Kafka?
    Yes. assign() opts out of group-managed partition assignment, but if group.id is set the consumer can still commit to __consumer_offsets (manually or via auto-commit). It just won't participate in rebalances.
  • What happens if you call both subscribe() and assign() on the same consumer?
    An IllegalStateException is thrown — topic subscription, pattern subscription, and manual assignment are mutually exclusive on a single consumer instance.

saying these in an interview costs you the question

  • Saying assign() automatically fails over to another instance — it does not.
  • Claiming assign() triggers rebalances — it never does.
  • Saying you can call subscribe() and assign() together to mix modes.
  • Thinking assign() means you cannot commit offsets at all.

context

resolving links…

open as a page

Explain the ConsumerRebalanceListener callbacks (onPartitionsRevoked, onPartitionsAssigned, onPartitionsLost) and what you should do in each.

level: seniorimportance: must knowfreq 60%

basics

~20 s

It's a callback you pass to subscribe() so you can react to rebalances: onPartitionsRevoked (commit offsets / flush state before losing partitions), onPartitionsAssigned (init state / seek for new partitions), onPartitionsLost (partitions gone unexpectedly — don't commit).

open as a page

With a consumer using assign() instead of subscribe(), how do you control where consumption starts, since there's no rebalance to trigger position setup?

level: middleimportance: should knowfreq 35%

basics

~10 s

After assign(), the consumer starts from the last committed offset for that group.id if one exists; otherwise auto.offset.reset (earliest/latest) applies. You can override explicitly with seek(), seekToBeginning(), or seekToEnd() before polling.

open as a page

How does pattern subscription (subscribe with a regex) work in Kafka, and what are its operational caveats?

level: middleimportance: should knowfreq 45%

basics

~10 s

subscribe(Pattern) matches topic names against a regex. The consumer auto-discovers new matching topics and rebalances to include them. It still uses group management, just with a dynamic topic set.

open as a page

You need deterministic, static partition ownership (each service instance always owns the same partitions) without rebalance pauses. How do you design this, and what do you give up?

level: principalimportance: should knowfreq 30%

basics

~20 s

Use assign() to pin a fixed partition set to each instance based on a stable ordinal (e.g. pod index). You avoid rebalances and get deterministic ownership, but you give up automatic load balancing and failover — you must handle scaling and dead-instance takeover yourself.

open as a page