How would a Rust crawler fetch 500 URLs with at most 16 in flight, handling each result as it arrives, using futures streams?
answer
- a stream of futures
- map is lazy
- a cap on in-flight futures
- completion order vs input order
basics
~10 sBuild stream::iter(urls).map(fetch).buffer_unordered(16) and loop with while let Some(r) = results.next().await. map creates each request future lazily, and buffer_unordered keeps at most 16 polled at once, yielding each result as it finishes.
solid answer
~40 s`futures::stream::iter(urls).map(fetch)` is a stream of futures: calling `fetch` only builds a request future, and only when the stream is polled. `.buffer_unordered(16)` pulls futures from that stream until 16 are held in its internal `FuturesUnordered`, polls them together, yields each output in completion order and pulls the next future as a slot frees up. Consuming it with `while let Some(page) = results.next().await` processes results as they land, with never more than 16 requests in flight. `buffered(16)` has the same cap but yields in input order, so one slow URL delays every result behind it. `join_all` over all 500 futures has no cap and returns only when every request is done. All of this is concurrency on one task, not parallelism.
code
rust · 30 linesuse futures::stream::{self, StreamExt};
use smol::Timer;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
static IN_FLIGHT: AtomicUsize = AtomicUsize::new(0);
static PEAK: AtomicUsize = AtomicUsize::new(0);
async fn fetch(url: String) -> (String, usize) {
let now = IN_FLIGHT.fetch_add(1, Ordering::SeqCst) + 1;
PEAK.fetch_max(now, Ordering::SeqCst);
Timer::after(Duration::from_millis(5 + (url.len() as u64 % 7))).await; // simulated request
IN_FLIGHT.fetch_sub(1, Ordering::SeqCst);
let len = url.len();
(url, len)
}
fn main() {
smol::block_on(async {
let urls: Vec<String> = (0..500).map(|i| format!("https://example.com/p/{i}")).collect();
let mut results = stream::iter(urls)
.map(fetch) // Stream of futures; nothing runs yet
.buffer_unordered(16); // at most 16 polled at once, yielded as they finish
let mut done = 0;
while let Some((_url, _len)) = results.next().await {
done += 1; // process each result as soon as it arrives
}
println!("done = {done}, peak in flight = {}", PEAK.load(Ordering::SeqCst));
});
}go deeper
Recall the shape stream::iter(urls).map(fetch).buffer_unordered(16) followed by a while let loop over next().await.
Explain that map only builds futures lazily, that buffer_unordered keeps at most n of them in a FuturesUnordered, and how buffered differs in output order.
Show you know where the bound leaks: slow awaits in the loop body stall the buffer, CPU work blocks the task, and concurrency is not a rate limit.
Relate the cap to downstream capacity: pick n from the remote side's limits and your memory budget, and measure rather than guess.
## The problem: bounded concurrency A crawler with 500 URLs has two bad extremes. Fetching them one at a time with `.await` in a loop wastes almost all its time waiting on the network. Firing all 500 at once can exhaust sockets, trip the remote server's rate limits and use far more memory than needed. The usual requirement is **bounded concurrency**: keep a fixed number of requests in flight, here 16, start a new one as each finishes, and handle each response as soon as it arrives. The `futures` crate's stream adapters express exactly that: ```rust let mut results = stream::iter(urls) .map(fetch) // a stream of futures; nothing runs yet .buffer_unordered(16); // at most 16 polled at once while let Some(page) = results.next().await { handle(page); } ``` ## How the pipeline works 1. **`stream::iter(urls)`** turns the `Vec<String>` into a stream that is always ready with the next URL. 2. **`.map(fetch)`** calls `fetch(url)` for each URL as it is pulled. An async fn call only builds a future, so this step creates request futures but sends nothing. 3. **`.buffer_unordered(16)`** requires the stream's items to be futures. Each time it is polled it pulls futures from upstream until 16 are in progress, pushes them into an internal `FuturesUnordered`, polls that set, and yields the output of whichever future finishes first. When a future finishes, a slot opens and the next poll pulls another URL. 4. **`while let Some(page) = results.next().await`** drives the whole thing; when upstream is exhausted and the last in-flight future finishes, `next()` returns `None` and the loop ends. The demonstration snippet counts concurrent requests with an atomic counter and reports `done = 500, peak in flight = 16`. ## Choosing the adapter | approach | in flight at once | order of results | results available | |---|---|---|---| | `.await` in a `for` loop | 1 | input order | one by one, slowly | | `join_all(urls.into_iter().map(fetch))` | all 500 | input order | only when all are done | | `.buffered(16)` | up to 16 | input order | in order; a slow item holds back later ones | | `.buffer_unordered(16)` | up to 16 | completion order | as each finishes | | `.for_each_concurrent(16, \|url\| ..)` | up to 16 | none (no output stream) | handled inside the closure | Points worth stating: - **`buffered` vs `buffer_unordered`.** Both cap in-flight futures at `n`; `buffered` reorders outputs to match the input, so one slow response blocks delivery of finished ones behind it (head-of-line blocking). - **`join_all`** has no limit at all. It suits a handful of futures, not a crawl; for large inputs (above an internal threshold of 30 in futures 0.3) it switches to `FuturesOrdered` internally, but it still starts every future at once. - **`for_each_concurrent(limit, f)`** is the shortcut when you only need side effects; it is a future that completes when every item has been handled. - **Errors.** If `fetch` returns `Result`, the loop receives each `Result` and decides; breaking out of the loop on an `Err` drops the stream, which drops and so cancels the requests still in flight. ## What the bound does not give you - **Parallelism.** The 16 requests share one task and one thread at a time. Waiting overlaps; CPU-heavy parsing inside `handle` does not, and while it runs no request is being polled. Move heavy parsing to threads or spawned tasks, which other lessons cover. - **Progress while the loop body awaits.** If `handle` itself awaits something slow, the buffer is not polled during that time, so the 16 in-flight requests stall. Keep per-result work short, or move it inside each future (`.map(|u| async move { handle(fetch(u).await).await })`). - **Rate limiting over time.** A cap on concurrency is not a cap on requests per second; fast responses can still produce many requests per second. In an interview, the one-liner is: a stream of lazily created futures, `buffer_unordered(n)` to cap how many are polled at once, and `while let Some(..) = next().await` to consume results in the order they complete.
- Why might a crawler deliberately choose `buffered(16)` over `buffer_unordered(16)`?When output order matters, for example writing results to a file that must match the input list, or pairing responses with the input without carrying the URL along. `buffered` keeps the same in-flight cap but delivers outputs in input order, at the cost of head-of-line blocking behind a slow request.
- What happens to the in-flight requests if the loop body awaits a slow database write for each result?While the body awaits, nothing polls the `buffer_unordered` stream, so the 16 in-flight futures make no progress past their current `.await` and no new requests start. Move the write inside each mapped future, or hand results to a separately driven consumer, so the slow work is polled concurrently too.
- Does `buffer_unordered(16)` limit requests per second?No. It limits how many futures are in progress at once. If responses come back in a few milliseconds, 16 concurrent slots can still produce thousands of requests per second; a time-based limit needs a rate limiter or timer between requests.
A coat check with 16 hooks: a new coat is taken only when a hook frees up, and coats go back to whoever returns first, not in the order they arrived.
- map(fetch) sends all 500 requests as soon as it is called
- buffer_unordered(16) runs the futures on 16 threads
- buffered and buffer_unordered differ in how many futures run at once
- join_all limits concurrency to a sensible default
- A concurrency cap also caps requests per second