Should KRaft controllers run on the same nodes as brokers, and why do production deployments isolate them?

## The three role configurations Every KRaft node sets `process.roles`: - `controller` — a dedicated voter: participates in the metadata Raft quorum, does not serve client produce/consume traffic. - `broker` — a dedicated data node: stores partitions, serves clients, fetches metadata from the controllers but does not vote. - `broker,controller` — **combined mode**: one process does both. ## Why combined mode exists Combined mode lets you stand up a working KRaft cluster on as few as one or three machines — ideal for laptops, CI, demos, and small non-critical clusters. It is fully functional, just operationally constrained. ## Why production isolates controllers ### 1. Resource isolation / interference Brokers do heavy I/O: large sequential writes, fetch requests serving consumers, page-cache churn, and occasionally long GC pauses under load. The controller's job — keeping the Raft metadata log healthy and sending/serving timely replication — is latency-sensitive. If a combined node is busy serving a traffic spike, the controller side can miss its `controller.quorum.fetch.timeout.ms` window, causing **spurious leader elections** or delayed metadata commits that ripple across the cluster. Dedicated controllers keep the control plane quiet and predictable. ### 2. Failure-domain separation With combined nodes, one machine failure removes a broker **and** a voter at once. In a 3-node combined cluster, losing one node simultaneously drops you to a 2-voter majority-of-2 (no remaining tolerance) and removes a data node. Separating roles means a broker loss doesn't touch quorum headroom and vice versa. You can also place the (few) controllers carefully across AZs without that constraint dictating broker placement. ### 3. Independent scaling Quorums don't scale with cluster size — you want 3 or 5 voters regardless of whether you have 5 brokers or 500. Dedicated controllers let you fix the quorum at 3/5 small nodes and scale brokers freely. In combined mode, every broker you add would (if also a controller) bloat the quorum, and if not, you'd have an awkward mix. ### 4. Operational safety Rolling restarts and upgrades: restarting a dedicated controller affects only quorum headroom briefly; restarting a combined node disrupts both planes. Role migration (e.g., separating roles later) is also cleaner when they started separate. Kafka's own guidance recommends dedicated controllers for production. ## Sizing the dedicated controllers Controllers need: modest CPU and memory, but a **fast, reliable disk** for the metadata log (it's on the commit path of every metadata change), and low-latency networking to each other and to brokers. They don't need the large disks brokers use for partition data. Run 3 (tolerate 1) or 5 (tolerate 2), spread across failure domains. ## Key takeaway Combined mode = dev/small. Production = dedicated controllers (3 or 5, isolated, AZ-spread) so data-plane load and failures can't destabilize the metadata quorum, and so you can scale and operate the two planes independently.