Explain the difference between data parallelism and task parallelism, give a concrete example of each, and say what determines how far each one can scale.
answer
- same code / different data vs different code / same time
- data parallel width grows with input; task parallel width is fixed
- task parallelism floor = critical path
- data parallelism = throughput; task parallelism = latency hiding
- they nest: parallel stages, parallel partitions inside a stage
basics
~20 sData parallelism runs the same operation over different slices of one data set — its scaling limit is how finely you can split the data. Task parallelism runs different operations concurrently — its scaling limit is the number of independent tasks and their dependency graph. Resizing a million images is data parallel; fetching a user profile and their orders at once is task parallel.
solid answer
~60 s**Data parallelism**: one operation, many data partitions. Split a collection into chunks, apply the same function to every chunk, combine. Example: computing a checksum for each of 10 million records, or resizing every image in a batch. The degree of parallelism is bounded by the number of partitions you can create — so it grows with the data, which is why it scales out across cores and machines. **Task parallelism**: different operations, run concurrently because they don't depend on each other. Example: a page render that fetches profile, orders and recommendations in parallel, then merges. The degree of parallelism is bounded by the number of independent tasks in the dependency graph — a fixed, usually small number that does *not* grow with input size. The critical path (longest dependency chain) sets the floor on completion time. They compose: a task-parallel stage can itself be internally data parallel. The practical consequence is that data parallelism is what you reach for when you want to use 64 cores, and task parallelism is what you reach for when you want to overlap a handful of independent latencies.
code
text · 13 linesDATA PARALLEL (width grows with N)
input: 1,000,000 records
split into 64 chunks -> 64 workers run score(chunk) -> merge counts
more records => more chunks => more usable cores
TASK PARALLEL (width fixed at 3)
render_page():
A: fetch_profile() |
B: fetch_orders() | independent, run concurrently
C: fetch_recs() |
D: merge(A, B, C) waits for all three
more users on the page => still 3 concurrent tasks
wall clock >= duration of the slowest of A/B/C, plus Dgo deeper
Define both with one clean example each, and state that data parallelism means the same operation on different slices of data.
Add the scaling argument: data-parallel width grows with input size, task-parallel width is capped by the number of independent tasks and the critical path.
Discuss how the two nest in a real pipeline, what each demands (associative combine and no shared accumulator vs a correct dependency graph), and which one you reach for depending on whether you lack CPU throughput or are hiding latency.
Frame decomposition as an architectural commitment — the partition key, state placement, rebalancing and failure granularity follow from the choice — and note that task decomposition sets a hard ceiling you cannot buy your way past.
## Two ways to decompose work When you want work to happen simultaneously, you must first decide *what* to split. There are two answers, and they behave very differently. **Data parallelism (domain decomposition).** The same computation is applied to different pieces of data. You partition the input, run identical code on each partition, and combine results. All workers execute the same instructions on different addresses. Examples: multiplying a matrix by splitting it into row blocks; scanning a 20 GB log by splitting it into byte ranges; scoring a million rows through the same model; adjusting brightness on every pixel. **Task parallelism (functional decomposition).** Different computations run at the same time because they are mutually independent. Workers execute *different* code. Examples: while rendering a dashboard, one task queries the database, another calls a pricing service, a third loads static config; in a compiler, parsing one file while type-checking another; in a game loop, running physics and audio concurrently within a frame. An easy test: if you doubled the input size, would you get more opportunities to run in parallel? If yes, it is data parallelism. If the number of concurrent things stays the same, it is task parallelism. ## Scaling behaviour, which is the point of the distinction **Data parallelism scales with data.** With N items and P workers you can, in principle, use every worker as long as N ≫ P. Add machines and you can add partitions. This is why every large-scale processing framework — parallel collections, GPU kernels, sharded databases, map-style batch jobs — is built on data parallelism. Its limits are partition skew (one partition much bigger than the others), the cost of splitting and combining, and any step that must see all data at once. **Task parallelism has a hard ceiling.** If the job decomposes into 5 independent tasks, 6 cores gain you nothing over 5, and no amount of extra hardware helps. Worse, tasks usually have dependencies: the merge step waits for the fetches. The **critical path** — the longest chain of dependent tasks — is a lower bound on wall-clock time no matter how many workers you have. Task parallelism therefore targets *latency hiding* (especially overlapping I/O waits), not throughput scaling. A useful summary: data parallelism gives you *scalable* parallelism whose width is a function of input size; task parallelism gives you *fixed-width* parallelism whose width is a function of program structure. ## They are not exclusive Real systems nest them. A request handler runs three independent service calls (task parallel); one of those calls internally re-ranks 50,000 candidates by splitting them across cores (data parallel). A stream processor runs distinct stages concurrently while each stage is replicated across partitions of the key space. The right question in an interview is not "which one is this?" but "where does the parallel width come from at each level, and what caps it?" ## What each one demands of your code **Data parallel** requires that per-partition work be independent: no partition may read another's in-progress results, and no shared mutable accumulator may be updated without coordination. The combine step must be associative (and commutative if partitions can finish in any order) or you must preserve partition order explicitly. The classic mistake is a shared counter or shared collection mutated from all workers — it either corrupts data or serialises the whole computation on one lock, erasing the speedup. **Task parallel** requires a correct dependency graph: you must know which tasks can start before which others finish. Errors show up as either a missing edge (a task reads a result that is not ready — a race) or an over-conservative edge (needless serialisation). Task-parallel work is also frequently heterogeneous in duration, so the slowest task dominates and load balancing is about *which* tasks, not *how many* items. ## Where the confusion usually lands Two neighbouring terms are worth separating explicitly: - **Concurrency vs parallelism.** Concurrency is a structuring property — multiple logical activities in flight, possibly interleaved on one core. Parallelism is a hardware property — activities physically executing at the same instant. Both decompositions above are ways of *obtaining* parallelism; task decomposition is also useful with no parallelism at all, purely to overlap waiting. - **Data parallelism vs SIMD.** SIMD (single instruction, multiple data — vector instructions) is a hardware realisation of data parallelism inside one core. Data parallelism as a design concept is independent of whether it is realised by vector lanes, threads, or machines. ## Choosing between them Ask what you are short of. If you are short of **CPU throughput** on a large input, decompose the data — that is the only route that scales with the machine. If you are short of **wall-clock time on a small input dominated by independent waits** (network, disk, other services), decompose the tasks — you are hiding latency, not adding compute. If the input is large *and* the pipeline has stages, do both: task-parallel stages, data-parallel within a stage.
- Your job splits into exactly four independent tasks and you are given a 32-core machine. What speedup can you expect, and what would you do about it?At most about 4x, and less if the tasks differ in duration, because the longest task bounds the wall clock. Extra cores are idle. To use them you must find data parallelism inside the tasks — partition the work each task does — or find more independent tasks by decomposing further. Adding hardware alone cannot exceed the task-graph width.
- Can a workload be both data parallel and task parallel at the same time?Yes, and large systems usually are. Distinct pipeline stages run concurrently as tasks, while each stage internally splits its input across workers as data. The useful analysis is per level: identify where the parallel width comes from at each layer and what caps it — task count and critical path at the outer level, partition count and skew at the inner one.
A restaurant kitchen. Ten cooks each chopping their own crate of onions is data parallelism — add more onions and you can add more cooks. One cook on sauce, one on grill, one plating is task parallelism — a fourth cook has no distinct station to take, and the meal is not ready until the slowest station finishes.
saying these in an interview costs you the question
- Using data parallelism and task parallelism as synonyms for multithreading in general.
- Claiming task parallelism scales with more cores, when its width is fixed by the number of independent tasks.
- Equating data parallelism with SIMD or vector instructions only; SIMD is one hardware realisation of the idea.
- Confusing concurrency (structure, interleaving) with parallelism (simultaneous execution on real hardware).
- Assuming data-parallel workers can freely share a mutable accumulator without coordination.