Your team's C# batch job calls an API for 10,000 ids with async code; what policy would you set for bounding concurrency, failures and fire-and-forget tasks?
answer
- Task.WhenAll bounds nothing
- a counting gate around each call
- the limit comes from downstream
- fail fast or collect outcomes
- every task has an owner
basics
~20 sNever fan 10,000 calls out unbounded: gate them with SemaphoreSlim or an equivalent limit set from the downstream's capacity, decide per job whether to fail fast or collect each outcome, put one deadline on the group, and ban untracked fire-and-forget tasks.
solid answer
~40 s`await Task.WhenAll(ids.Select(CallApiAsync))` starts all 10,000 requests at once, because `Task.WhenAll` limits nothing; the result is connection exhaustion, throttling responses and memory spikes. My policy has four parts. **Bound it:** a `SemaphoreSlim(n)` with `await gate.WaitAsync(token)` before each call and `gate.Release()` in `finally`, or `Parallel.ForEachAsync` with `MaxDegreeOfParallelism`; `n` comes from the API's rate and connection limits, is configurable and is measured, not `Environment.ProcessorCount`. **Choose the failure mode:** fail fast with a shared cancellation source, or return a per-id result so one 503 does not discard 9,999 successes. **One deadline** for the group through a `CancellationTokenSource` with a timeout. **No untracked tasks:** every task is awaited or owned, and CS4014 is not suppressed.
code
csharp · 30 linesint[] ids = Enumerable.Range(1, 10_000).ToArray();
using var deadline = new CancellationTokenSource(TimeSpan.FromMinutes(5));
using var gate = new SemaphoreSlim(32); // at most 32 calls in flight
async Task<(int Id, int? Value, Exception? Error)> FetchAsync(int id)
{
await gate.WaitAsync(deadline.Token);
try
{
return (id, await CallApiAsync(id, deadline.Token), null);
}
catch (HttpRequestException ex)
{
return (id, null, ex); // record it, keep the batch going
}
finally
{
gate.Release();
}
}
var results = await Task.WhenAll(ids.Select(FetchAsync));
Console.WriteLine($"ok {results.Count(r => r.Error is null)}, failed {results.Count(r => r.Error is not null)}");
static async Task<int> CallApiAsync(int id, CancellationToken token)
{
await Task.Delay(1, token);
if (id % 1000 == 0) throw new HttpRequestException($"id {id}: 503");
return id * 2;
}go deeper
Recall that Task.WhenAll starts nothing and limits nothing, so a large list of calls needs an explicit concurrency limit such as SemaphoreSlim.
Explain the WaitAsync and Release-in-finally pattern, why the gate is a slot counter rather than a lock, and how chunking differs from a sliding window.
Diagnose socket exhaustion or throttling from unbounded fan-out, choose fail-fast or per-item results for the job, and add one deadline for the group.
Own the rule: how limits are derived from downstream capacity and made configurable, which failure mode each job type uses, and how untracked tasks are banned and detected.
## Why this is a lead's decision A batch that calls an API for 10,000 ids can be written correctly in several ways, and they behave very differently under load and failure. The choice of limit, failure mode and ownership rules affects other teams' services, so it belongs in a shared rule rather than in each developer's head. ## The unbounded version ```csharp int[] results = await Task.WhenAll(ids.Select(id => CallApiAsync(id))); ``` `Task.WhenAll` does not throttle: `Select` calls `CallApiAsync` for every id as `WhenAll` enumerates it, so all 10,000 requests are in flight almost at once. Typical consequences: - the HTTP client's connection pool and the machine's sockets run out; - the downstream answers with throttling or timeouts, and every caller suffers; - 10,000 pending tasks, buffers and responses sit in memory together; - one failure makes the whole `await` throw, discarding every success. ## Option 1: a counting gate ```csharp using var gate = new SemaphoreSlim(32); async Task<Result> FetchAsync(int id) { await gate.WaitAsync(token); try { return await CallApiAsync(id, token); } finally { gate.Release(); } } var results = await Task.WhenAll(ids.Select(FetchAsync)); ``` `SemaphoreSlim` here is a **counter of free slots**, not a lock around shared state. `WaitAsync` waits without blocking a thread, and `Release` in `finally` guarantees a failed call returns its slot; a missed release shrinks the pool permanently. This version still creates 10,000 small tasks that queue on the gate; for very large inputs, acquire the slot in the loop that starts the calls instead, so only `n` exist at a time. ## The alternatives | Approach | Strength | Weakness | |---|---|---| | `SemaphoreSlim` gate + `Task.WhenAll` | simple, keeps `n` in flight | all task objects created up front | | fixed chunks, `Task.WhenAll` per chunk | trivial to read | each chunk waits for its slowest call, so slots sit idle | | `Parallel.ForEachAsync` with `MaxDegreeOfParallelism` | built-in, cancellable | its scheduling is covered with the thread pool material | | a producer/consumer queue | back-pressure across stages | more moving parts than one batch needs | ## Choosing the limit The number is a **capacity** decision, not a CPU one. I/O-bound calls barely use the CPU, so `Environment.ProcessorCount` is the wrong input. Inputs that matter: 1. the downstream's published rate limit and how many of our instances share it; 2. the connection limit per server in the HTTP client; 3. observed latency, since throughput is roughly limit divided by latency; 4. what the batch can afford to occupy while user traffic shares the same dependency. Make the limit configuration, start conservative and raise it with measurements. ## Failure policy Pick one explicitly per job: - **Fail fast:** all calls share a `CancellationTokenSource`; the first failure cancels it, the remaining calls end canceled, and the combined task ends faulted with that first failure, because a fault outranks cancellations. Right when partial output is useless. - **Collect outcomes:** each call catches its expected failures and returns a per-id result, as the example below does, so 9,990 successes survive 10 errors and the failures can be retried later. Retry policy itself belongs with resilience tooling. Either way, add **one deadline** for the whole group, such as `new CancellationTokenSource(TimeSpan.FromMinutes(5))`, passed to `WaitAsync` and to every call, so a stalled dependency cannot hold the job open indefinitely. ## Fire-and-forget rules - No `_ =` on a task inside batch code; every task is awaited or registered with an owner that logs faults and drains at shutdown. - CS4014 is never suppressed with a pragma; the warning marks exactly this risk. - A process-wide `TaskScheduler.UnobservedTaskException` handler logs anything that slips through, and each hit is treated as a bug. ## Making it observable A limit nobody can see gets tuned by guesswork. Record how long calls wait at the gate, how many are in flight and how many fail per error type; a wait time that dominates the call time means the limit, not the API, sets the batch's speed. ## The trade-off to articulate There is no single right number or mode. A lead weighs batch speed against the shared dependency's health, all-or-nothing semantics against partial progress, and simplicity against memory use, then writes the choice down so the next job does not rediscover it in production.
- Why not split the ids into chunks of 32 and await `Task.WhenAll` on each chunk?Each chunk finishes only when its slowest call does, so while one straggler runs, 31 slots sit idle and throughput drops toward the slowest latency. A gate keeps 32 calls in flight continuously, starting a new one the moment any slot frees.
- How would you make the batch fail fast instead of collecting failures?Give every call a token from one shared `CancellationTokenSource`, catch the first failure inside the per-id wrapper, cancel the source and rethrow. The remaining calls stop at their next token check and end canceled, so the combined task ends faulted with the original error; log that, not the cancellations.
- When is untracked fire-and-forget acceptable at all?Microsoft's guidance names three conditions together: the work is optional, its failure is safe to ignore, and it finishes well within the process lifetime, such as a non-critical telemetry ping. Even then, attaching a continuation that logs faults costs little.
- Task.WhenAll limits how many tasks run at the same time.
- The concurrency limit for I/O calls should equal Environment.ProcessorCount.
- Releasing the SemaphoreSlim outside a finally block is fine.
- Starting all 10,000 calls at once is fastest because the I/O is async.
- Each id needs its own Task.Run for the calls to overlap.