An ops team complains their stateful Kafka Streams aggregation can't keep up despite adding more machines. Throughput plateaus. Diagnose the likely cause and the remediation, including the risks.

## Diagnosis: the partition-count ceiling Kafka Streams parallelism is bounded by **input partition count**. The aggregating sub-topology has **one task per partition** of its input (or of the **repartition topic** that feeds the `groupBy`/aggregate). Total worker slots = sum of `num.stream.threads` across all instances. Once **slots ≥ tasks**, every extra slot is **idle** — so adding machines yields no throughput gain. A flat throughput curve as you scale out is the classic signature. Secondary cause: **partition skew / hot keys**. Even below the ceiling, if a few keys dominate, the partitions holding them become the bottleneck while others idle. One overloaded task caps end-to-end lag regardless of total capacity. ### How to confirm - Compare the input/repartition topic's **partition count** to the number of **active tasks** (and to total threads). If threads ≥ partitions, you're ceiling-bound. - Inspect per-partition consumer lag; uneven lag points to **skew**, even lag at full thread utilization points to the **ceiling**. - Check CPU: ceiling-bound idle instances show low CPU; skew shows one instance/thread hot. ## Remediation ### Option A — increase input partition count More partitions → more tasks → more parallelism. **The catch**: partitions are chosen by `hash(key) % numPartitions`. Changing `numPartitions` **re-routes existing keys to different partitions**. Consequences: - **Per-key ordering** across the boundary is broken (a key's old and new records may sit in different partitions). - **State stores** are partitioned by the old scheme; their changelog data no longer lines up with the new partition→task mapping. The local state is effectively invalid. - You typically must **stop the app, reset it** (`kafka-streams-application-reset`), repartition the topic, and **rebuild state** from source — a planned migration with downtime or a parallel-run cutover. - Upstream producers must also adopt the new partition count consistently. ### Option B — fix skew If the issue is hot keys, increasing partitions won't help (the hot key still goes to one partition). Instead: introduce a **composite/sub-key** (salting) to spread a hot key across partitions, then re-aggregate, or pre-aggregate upstream. ### Option C — vertical headroom / tuning If instances are under-threaded, raise `num.stream.threads` up to the task count first (cheap, no repartition). Tune RocksDB, batching, and `commit.interval.ms` to raise per-task throughput. ## The durable lesson (principal-level) Because repartitioning is so disruptive, **partition count is a capacity-planning decision made before launch**. Pick input partition counts that leave scaling headroom (e.g. a multiple of expected max instances), accepting some per-partition overhead, so you can scale out later without a re-key migration. This is why teams 'over-partition' high-throughput topics deliberately. ## Edge cases - **Stateless** pipelines tolerate repartitioning better (no state to rebuild) but still break per-key ordering. - **Standby replicas / more threads** never raise the ceiling — they're availability and CPU utilization, not added tasks. - A **repartition topic** deep in the topology imposes its own ceiling; sometimes you can bump *its* partition count (via `Repartitioned.numberOfPartitions`) without touching the source, but co-partitioning constraints with joined streams still apply.