How does tasks.max interact with worker count when balancing load, and what are the failure modes of misconfiguring it?

## What tasks.max is `tasks.max` is a **per-connector** configuration: the maximum number of **tasks** the framework will let that connector create. Recall that a **task** is the actual unit of data-copying work and the unit the Connect leader **balances across workers** during assignment. So `tasks.max` is effectively your lever on a connector's parallelism. ## Effective parallelism = min(tasks.max, splittable work) Setting `tasks.max=10` does **not** guarantee 10 tasks. The connector decides how to split its job, and it can't manufacture parallelism that doesn't exist: - A **source** connector copying a single, non-partitioned database table may only ever produce **1 task**, no matter how high tasks.max is. - A **source** connector watching 12 tables (or a 12-partition log) can produce up to 12 tasks. - A **sink** connector consuming a topic is bounded by the topic's **partition count**: you can't usefully have more sink tasks than partitions, because a partition is consumed by at most one task. So: **effective tasks = min(tasks.max, what the connector can split)**. ## How it interacts with worker count The leader spreads a connector's tasks across the workers. Therefore: - The number of workers that can help a connector ≤ its effective task count. - For even balancing, you want effective tasks ≥ worker count (ideally a multiple), so each worker gets a fair share. ## Failure mode 1: tasks.max too low If effective tasks < worker count: - **Idle workers**: workers with no task for this connector do nothing for it — wasted capacity you're paying for. - **Throughput ceiling**: the connector can't go faster than its few tasks allow; adding workers won't help. - Example: 8 workers, `tasks.max=2` on your highest-volume connector → only 2 workers do that work; you're leaving 6 workers' worth of capacity on the table. ## Failure mode 2: tasks.max too high If tasks.max far exceeds useful work: - **No extra throughput**: you can't parallelize beyond the splittable work; extra tasks are either not created or idle. - **Overhead**: more tasks = more connections/threads to the external system, more state for the leader to assign, larger/slower rebalances. - **Resource exhaustion**: each task often opens its own connection (DB session, HTTP client). A huge tasks.max can blow through the source/sink system's connection or thread limits, or oversubscribe a worker's CPU/memory. - **Oversubscription per worker**: piling many tasks onto few workers can starve each task of resources, hurting throughput rather than helping. ## The balanced target Size `tasks.max` to: 1. The connector's **splittable granularity** (partitions, tables, files) — there's no point exceeding it. 2. Roughly the **worker count** (or a small multiple), so each rebalance can distribute tasks evenly and adding a worker actually offloads work. 3. The **downstream system's** capacity (connection pools, write throughput) — more tasks hammer it harder. ## Rebalancing connection Because tasks are what get moved during a rebalance, a well-chosen `tasks.max` also makes incremental cooperative rebalancing effective: with enough tasks, the leader can shift a few onto a newly joined worker for smooth scale-out. With too few tasks, a new worker may get nothing until you raise tasks.max (which itself triggers a rebalance, since changing tasks.max reconfigures the connector). ## Gotcha: changing tasks.max triggers a rebalance Editing `tasks.max` is a connector reconfiguration, so it causes the connector's tasks to be recomputed and reassigned — under cooperative rebalancing this is incremental and cheap, but it's not free.