In Rust with Rayon, how do `par_iter`, `par_iter_mut` and `into_par_iter` parallelise an iterator chain, and what does each one borrow?
answer
- change the first call only
- trait methods need the prelude
- shared refs, mutable refs, owned values
- Sync for &T, Send for the rest
basics
~20 sRayon's par_iter, par_iter_mut and into_par_iter replace iter, iter_mut and into_iter, returning a ParallelIterator that splits items across a global thread pool. They yield &T (needs T: Sync), &mut T or owned T (both need T: Send).
solid answer
~40 sRayon parallelises a chain by swapping its first call: `iter()` becomes `par_iter()`, `iter_mut()` becomes `par_iter_mut()`, `into_iter()` becomes `into_par_iter()`, after `use rayon::prelude::*` brings the traits into scope (without it rustc reports E0599). The result is a `ParallelIterator` with familiar adapters such as `map`, `filter` and `sum`, executed on Rayon's global pool, one worker per logical CPU by default. `par_iter` hands out `&T` to many threads, so `T: Sync`; `par_iter_mut` hands each `&mut T` to one worker and `into_par_iter` moves each `T`, so both need `T: Send`. Closures must be `Fn + Sync + Send` but may borrow locals, because the call blocks until every item is done.
code
rust · 24 linesuse rayon::prelude::*;
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
fn digest(bytes: &[u8]) -> u64 {
let mut h = DefaultHasher::new();
bytes.hash(&mut h);
h.finish()
}
fn main() {
let mut files: Vec<Vec<u8>> = (0..1_000u32).map(|i| i.to_le_bytes().repeat(256)).collect();
// &Vec<Vec<u8>> -> items are &Vec<u8>; needs Vec<u8>: Sync
let digests: Vec<u64> = files.par_iter().map(|f| digest(f)).collect();
// &mut Vec<Vec<u8>> -> items are &mut Vec<u8>; needs Vec<u8>: Send
files.par_iter_mut().for_each(|f| f.truncate(16));
// Vec<Vec<u8>> by value -> items are Vec<u8>; the Vec is consumed
let total: usize = files.into_par_iter().map(|f| f.len()).sum();
println!("{} digests, first {:x}, {} bytes kept", digests.len(), digests[0], total);
}go deeper
Recall the three renamed entry points, the prelude import, and that the chain keeps the same adapters. Say which one consumes the collection.
Explain why par_iter needs T: Sync while par_iter_mut and into_par_iter need T: Send, and why closures must be Fn + Sync + Send but may borrow locals.
Read an E0599 on par_iter as an unmet Send or Sync bound on the element type, and justify switching only after a release-build measurement on CPU-heavy items.
Treat par_iter as a local optimisation with a measurable cost; push teams to keep sequential and parallel versions comparable and to parallelise only proven CPU hot spots.
## What Rayon adds to an iterator chain **Rayon** is the data-parallelism crate most Rust code reaches for when a loop over a collection is **CPU-bound**: every item needs real computation and the items do not depend on each other. Its central idea is that a parallel chain looks like the sequential one. You replace the call that starts the chain and keep the adapters: - `v.iter()` becomes `v.par_iter()` - `v.iter_mut()` becomes `v.par_iter_mut()` - `v.into_iter()` becomes `v.into_par_iter()` What comes back is not a `std::iter::Iterator` but a type implementing Rayon's own **`ParallelIterator`** trait, and for slices, vectors and ranges also **`IndexedParallelIterator`**. Together they offer most of the adapters you know (`map`, `filter`, `flat_map`, `zip`, `enumerate`) and consumers (`collect`, `sum`, `for_each`, `any`, `count`, `min`), plus parallel-only ones such as `fold` that returns one partial result per split, `reduce` and `find_any`. The methods live on traits, so they are only callable when the traits are in scope. The crate's README recommends one line per module that uses them: ```rust use rayon::prelude::*; ``` Forget it and rustc reports **E0599**, ``no method named `par_iter` found for struct `Vec<u64>` in the current scope``, and helpfully suggests `iter` instead. ## The three entry points and what each one borrows | Call | Implemented for | Item type | Bound Rayon places on `T` | |---|---|---|---| | `par_iter()` | `&[T]`, `&Vec<T>` and other shared borrows | `&T` | `T: Sync` | | `par_iter_mut()` | `&mut [T]`, `&mut Vec<T>` | `&mut T` | `T: Send` | | `into_par_iter()` | `Vec<T>`, ranges, `Option`, owned collections | `T` | `T: Send` | The bounds follow from what crosses threads. `par_iter` hands **shared references** to several worker threads at once, so `T` must be `Sync` (a `&T` is `Send` exactly when `T` is `Sync`). `par_iter_mut` hands each `&mut T` to exactly one worker, which needs only `T: Send`. `into_par_iter` **moves** the values into the workers, which also needs `T: Send`, and consumes the collection, so the `Vec` is unusable afterwards. When the element type fails the bound, rustc does not even find the method: `par_iter()` on a `Vec<Rc<str>>` fails with E0599, "the method `par_iter` exists for struct `Vec<Rc<str>>`, but its trait bounds were not satisfied". The reason `Rc` is not thread-safe belongs to the material on reference counting; the point here is that Rayon's entry points carry `Send` and `Sync` bounds and the compiler enforces them. ## The closures and the pool Adapters take closures with stricter bounds than their sequential twins. `ParallelIterator::map` requires `F: Fn(Self::Item) -> R + Sync + Send`, and every item type is `Send`. In practice that means: 1. The closure may **borrow** locals: `files.par_iter().map(|f| digest(f))` captures nothing, and a closure that reads a borrowed lookup table compiles, because Rayon's calls block until all work is done and need no `'static`. 2. The closure is called through `&`, many times, from several threads, so it cannot mutate captured state; that is a lesson of its own. 3. Results come back the way the sequential chain would produce them: a `collect` into `Vec` keeps the input order. The work runs on Rayon's **global thread pool**, created lazily on first use with one worker per logical CPU unless the `RAYON_NUM_THREADS` environment variable says otherwise. The calling thread blocks until the parallel chain has finished, so from the outside a `par_iter` chain behaves like an ordinary function call that happens to use every core. ## When the switch pays off Parallelism has a cost: splitting the input, scheduling tasks and combining results. It pays when each item does meaningful CPU work, such as hashing a file's contents, parsing, compressing or simulating. It does not pay for summing a few hundred integers, where the sequential loop wins, or for work that mostly waits on disks or the network, which blocks Rayon's workers. Measure both versions in a release build before committing to either. A few conversion slips come up repeatedly in review: - **Starting from an iterator instead of a collection**: `v.iter().par_iter()` fails with E0599, because `par_iter` exists for borrowed collections, not for `std::slice::Iter`. Start from `v`, or call `par_bridge()` on an existing `Iterator` and accept that it pulls items one at a time and does not keep their order. - **Expecting a std `Iterator` back**: a parallel chain ends in a Rayon consumer such as `collect`, `sum` or `for_each`; it cannot be passed to a function that takes `impl Iterator`. - **Sequential state in the closure**: a running counter or a `push` into a captured `Vec` does not compile, because the closure must be `Fn`; return a value and let the consumer combine them.
- Why does `par_iter()` on a `Vec<Rc<str>>` fail with E0599 rather than E0277?`par_iter` comes from a blanket trait implemented only when `&Vec<T>` implements `IntoParallelIterator`, which requires `T: Sync`. `Rc<str>` is not `Sync`, so the method is simply not available and rustc says it exists but its trait bounds were not satisfied. Capturing an `Rc` inside a `map` closure is different: there the method resolves and the closure's `Sync` bound fails with E0277.
- Can the closure passed to a Rayon `map` read a local `HashMap` without wrapping it in `Arc`?Yes. Rayon's consumers such as `collect` and `sum` block until every item is processed, so the closure needs no `'static` bound and can capture `&HashMap` directly. The closure must be `Sync` and `Send`, which a shared reference to a `Sync` map satisfies. An `Arc` is only needed when work outlives the borrow, as with a detached `rayon::spawn`.
- par_iter works without importing anything from Rayon
- par_iter returns a std Iterator you can pass to std APIs
- into_par_iter leaves the Vec usable after the chain
- Every Rayon closure must be 'static like a thread::spawn closure
- par_iter only needs the element type to be Send