skip to content

When does Rake's multitask speed up a build, and what races can running prerequisites in parallel introduce?

level: seniorimportance: should knowfreq 20%

answer

  1. immediate prerequisites in threads
  2. shared prerequisites still run once
  3. -j jobs, default cores + 4
  4. sh subprocesses really overlap
  5. -m makes every task a multitask

basics

~20 s

Rake's multitask runs a task's immediate prerequisites in parallel on a thread pool sized by -j. It helps when they shell out or wait on I/O; shared prerequisites still run once, but Ruby data they share needs synchronisation.

solid answer

~40 s

`multitask docs: %w[docs:api docs:guides docs:changelog]` invokes those prerequisites concurrently on Rake's thread pool, then runs `docs`'s own actions once all finish. Rake's own bookkeeping is thread-safe: each task invokes under a lock, so a prerequisite shared by several parallel branches runs once and the others wait for it. Speed-up comes when prerequisites run external commands through `sh` or wait on I/O; pure Ruby computation mostly takes turns under CRuby's global VM lock. The pool size is `-j` (`--jobs`), defaulting to the CPU count plus four; `-m` treats every task as a multitask. The races are yours: two branches writing the same file, appending to a shared Array, calling `Dir.chdir`, or relying on an ordering that `task` guaranteed but `multitask` does not. If one branch fails, dependents re-raise that same exception.

code

ruby · 16 lines
ruby
task :prepare do
  mkdir_p "pkg"
end

file "pkg/ledger_kit.gem" => :prepare do
  sh "gem", "build", "ledger_kit.gemspec", "-o", "pkg/ledger_kit.gem"
end

file "pkg/docs.tar.gz" => :prepare do
  sh "tar", "czf", "pkg/docs.tar.gz", "doc"
end

desc "Build release assets in parallel"
multitask assets: %w[pkg/ledger_kit.gem pkg/docs.tar.gz] do
  puts "assets ready"
end

go deeper

for a junior

Know that multitask runs a task's prerequisites in parallel and that shared prerequisites still run once.

for a middle

Explain the thread pool, -j and -m, and why subprocess-heavy branches gain while pure Ruby branches may not.

for a senior

Find and fix ordering and shared-state races before enabling parallelism, declaring true dependencies instead of relying on list order.

for a principal

Weigh build parallelism against reproducibility and debuggability, and decide where parallel builds belong: Rake, the CI runner or separate jobs.

## What multitask does **`multitask`** is declared exactly like `task`, but its **immediate prerequisites** are invoked in parallel instead of one after another: - `multitask release_assets: %w[pkg/app.gem docs/index.html checksums.txt]` Each prerequisite is submitted to Rake's **thread pool** as a future; the multitask's own actions run after all of them complete. Prerequisites of those prerequisites are invoked by whichever thread gets there first. ## What stays safe Rake's documentation states its internal data structures are thread-safe for multitask execution, and the task code backs it up: each task's invocation runs under a per-task lock. - A **shared secondary prerequisite** — say `prepare` needed by all three branches — runs **once**; the other threads block on its lock and then see it as already invoked. - If that shared task **fails**, every branch that needs it re-raises the same stored exception rather than retrying. - Circular dependencies are still detected. ## When it is faster Parallelism only helps when the branches can actually overlap: | Work in each branch | Expected effect | |---|---| | `sh` commands (compilers, `gem build`, doc generators) | real overlap: subprocesses run on separate cores | | network or disk waits | overlap while threads wait | | pure Ruby computation | little gain on CRuby: threads mostly take turns under the global VM lock | | a single long branch | no gain: the total is bounded by the slowest branch | ## Controlling the pool 1. **`-j N`** (`--jobs`) caps parallel tasks; without it the pool is sized from the CPU count plus four. A bare `-j` removes the cap. 2. **`-m`** (`--multitask`) treats **every** task as a multitask for that run — a quick experiment that also exposes ordering assumptions. 3. **`--job-stats`** prints statistics about the thread pool after the run. ## The races you own Rake protects its own state, not yours. Common failures: - **Shared output**: two branches writing the same file, or both running `rm_rf "tmp"`. - **Shared Ruby objects**: appending to a constant Array or Hash from several branches without a `Mutex`. - **Process-wide state**: `Dir.chdir` (Rake's `chdir` helper included) changes the directory for every thread; `ENV` changes are global too. - **Hidden ordering**: with `task`, prerequisites run left to right, so `%w[clean build]` happened to work; with `multitask` both start together and `build` may read what `clean` is deleting. Express real ordering as a dependency (`task build: :clean`) instead of list position. - **Interleaved output**: logs from branches mix, which makes failures harder to read. ## A reasonable approach - Keep branches independent: separate output paths, no shared mutable Ruby objects. - Make every real ordering a declared prerequisite. - Run `rake -m` once in CI to flush out hidden ordering, then decide which tasks should really be multitasks. - Measure; if branches are pure Ruby, processes (for example via `sh` running separate `ruby` commands) parallelise better than threads.

  • Two branches of a multitask both depend on prepare. Does prepare run twice?
    No. Each task invokes under its own lock and marks itself invoked, so the first thread runs `prepare` and the others wait, then find it already invoked and continue. If `prepare` raised, they all re-raise that stored exception.
  • Your Rakefile says task release: %w[clean build]. Why might it break under rake -m?
    With `task`, prerequisites run left to right, so `clean` finished before `build` by accident of list order. `-m` turns every task into a multitask, so `clean` and `build` start together and `build` may read files `clean` is deleting. Declare the real ordering: `task build: :clean`.
  • Why might a multitask of pure-Ruby prerequisites barely speed up on CRuby?
    Multitask uses threads, and on CRuby only one thread runs Ruby code at a time under the global VM lock. Threads overlap while waiting on subprocesses or I/O, so `sh`-heavy branches parallelise well, while CPU-bound Ruby branches mostly take turns.

saying these in an interview costs you the question

  • multitask runs a shared secondary prerequisite once per branch.
  • multitask runs its own actions in parallel with its prerequisites.
  • Rake makes shared Ruby Arrays and Hashes thread-safe automatically.
  • Prerequisite list order still sets execution order under multitask.
  • multitask always speeds up CPU-bound Ruby tasks on CRuby.