In .NET, what do BlockingCollection<T>'s bounded capacity, CompleteAdding and GetConsumingEnumerable each do in a producer–consumer pipeline?
answer
- a wrapper over a concurrent collection
- back-pressure on producers
- marking the end of input
- a foreach that removes items
basics
~20 sBlockingCollection<T> wraps a ConcurrentQueue<T> by default. A bounded capacity makes Add block while it is full, Take blocks while it is empty, and CompleteAdding marks the end so GetConsumingEnumerable finishes once the remaining items are taken.
solid answer
~40 s`BlockingCollection<T>` adds blocking and bounding to any `IProducerConsumerCollection<T>`: a `ConcurrentQueue<T>` unless you pass a `ConcurrentStack<T>` or `ConcurrentBag<T>` to the constructor. With `new BlockingCollection<T>(boundedCapacity: 2)`, `Add` blocks the producing thread while two items are waiting, which is back-pressure. `Take` blocks while it is empty, and `TryAdd`/`TryTake` accept a timeout and a `CancellationToken`. `CompleteAdding()` declares that no more items will come: a later `Add` throws `InvalidOperationException`, and `GetConsumingEnumerable()`, a sequence that removes each item as it yields it, ends once the collection is marked complete and empty. Forget `CompleteAdding` and the consumer waits forever. The type holds `SemaphoreSlim` instances, so dispose it. It blocks real threads, which suits dedicated worker threads; asynchronous code uses channels instead.
code
csharp · 24 linesusing System.Collections.Concurrent;
// At most 2 pending log batches; a producer that gets ahead blocks in Add.
using var batches = new BlockingCollection<string>(boundedCapacity: 2);
var consumer = Task.Run(() =>
{
foreach (string batch in batches.GetConsumingEnumerable())
{
Thread.Sleep(20); // stands in for a slow upload
Console.WriteLine($"uploaded {batch}");
}
Console.WriteLine("consumer done");
});
for (int i = 1; i <= 4; i++)
{
batches.Add($"batch-{i}"); // blocks while 2 batches are already waiting
}
batches.CompleteAdding(); // lets GetConsumingEnumerable finish once empty
await consumer;
try { batches.Add("late"); }
catch (InvalidOperationException e) { Console.WriteLine($"InvalidOperationException: {e.Message}"); }go deeper
Know that BlockingCollection<T> lets consumers wait for items, and that CompleteAdding tells them no more are coming.
Explain bounded capacity as back-pressure, the difference between IsAddingCompleted and IsCompleted, and why GetConsumingEnumerable is the consumer loop.
Catch the hangs and crashes: a missing CompleteAdding, an Add after completion, a plain foreach that never consumes, and blocked thread-pool threads in async code.
Decide where blocking hand-offs belong: dedicated worker threads may keep BlockingCollection<T>, while async services standardise on channels with explicit bounds and drop policies.
## What BlockingCollection<T> adds The concurrent collections never wait: `ConcurrentQueue<T>.TryDequeue` returns `false` on an empty queue, and nothing stops a producer from filling memory. `BlockingCollection<T>` (in `System.Collections.Concurrent` since .NET Framework 4) wraps an `IProducerConsumerCollection<T>` and adds two things: - **Blocking:** a consumer that calls `Take` sleeps until an item arrives, instead of polling. - **Bounding:** an optional maximum size; a producer that calls `Add` while it is full sleeps until a slot frees up. Internally it holds two `SemaphoreSlim` counters, one for occupied slots and, when bounded, one for free slots. That is also why it implements `IDisposable`. | Constructor | Backing collection | Bound | |---|---|---| | `new BlockingCollection<T>()` | `ConcurrentQueue<T>` (FIFO) | none | | `new BlockingCollection<T>(int boundedCapacity)` | `ConcurrentQueue<T>` | `boundedCapacity` | | `new BlockingCollection<T>(IProducerConsumerCollection<T>)` | the one you pass | none | | `new BlockingCollection<T>(IProducerConsumerCollection<T>, int)` | the one you pass | `boundedCapacity` | Passing a `ConcurrentStack<T>` turns it into a blocking LIFO; the order always comes from the wrapped collection. ## Bounding is back-pressure When a fast producer feeds a slow consumer, an unbounded queue just grows. With a bound: 1. `Add(item)` waits on the free-slot semaphore, so the producer stops when the collection is full. 2. `Take()` removes an item and releases a free slot, waking one blocked producer. 3. `TryAdd(item, millisecondsTimeout)` and `TryAdd(item, TimeSpan)` give up after the timeout and return `false`, so a producer can shed load instead of waiting forever. 4. Overloads that take a `CancellationToken` throw `OperationCanceledException` when it is cancelled. `BoundedCapacity` returns the bound, or -1 when there is none. ## Ending the stream A consumer needs to know when input is over, which an empty collection alone cannot tell it. `CompleteAdding()` provides that signal: - `IsAddingCompleted` becomes `true` at once; any later `Add` throws `InvalidOperationException` ("The collection has been marked as complete with regards to additions."). - `IsCompleted` becomes `true` when adding is complete **and** the collection is empty. - `Take()` on a completed collection throws `InvalidOperationException`; `TryTake` returns `false`. - `GetConsumingEnumerable()` returns a sequence that calls a blocking take on each step and stops when the collection is completed, so a plain `foreach` over it is the whole consumer loop. Two traps sit here. First, a producer that never calls `CompleteAdding` leaves the consumer's `foreach` blocked forever once the items run out; call it in a `finally` after producing. Second, a `foreach` directly over the `BlockingCollection<T>` (not over `GetConsumingEnumerable()`) uses the wrapped collection's own enumerator: for the default queue that is a snapshot, which neither removes items nor waits for new ones. ## Several consumers, and disposal Several threads may iterate `GetConsumingEnumerable()` on the same collection at once; each item goes to exactly one of them, which spreads work across a fixed set of workers. After `Dispose`, members such as `Add`, `Take`, `BoundedCapacity` and `GetConsumingEnumerable` throw `ObjectDisposedException`, so dispose only once producers and consumers have finished. ## Compared with polling a ConcurrentQueue<T> Without the wrapper, a consumer of a bare `ConcurrentQueue<T>` has to poll: - **A tight `TryDequeue` loop** burns a CPU core while the queue is empty. - **A loop with `Thread.Sleep`** wastes less CPU but adds up to one sleep interval of latency to every item. - **No bound:** nothing slows producers down, so a stalled consumer turns into growing memory. - **No end signal:** an empty queue cannot say whether more items are coming, so the code needs a separate flag, with its own race between the flag and the queue. `BlockingCollection<T>` solves all four with semaphores, `BoundedCapacity` and `CompleteAdding`. ## Where it fits today `BlockingCollection<T>` blocks **threads**. That suits a dedicated worker thread or a console tool. In asynchronous server code, a blocked thread-pool thread is a thread taken away from other requests, so an async producer–consumer design uses channels, which have their own material. In the API gateway, a background thread that uploads log batches can use a bounded `BlockingCollection<LogBatch>`: request threads call `TryAdd` with a short timeout and drop a batch rather than stall a request when the uploader falls behind.
- What happens if the producer never calls CompleteAdding?The consumer's `foreach` over `GetConsumingEnumerable()` takes every item and then blocks, waiting for one that never comes, because only `CompleteAdding` lets the sequence end. The program hangs at shutdown. Call `CompleteAdding` in a `finally` once production ends, or pass a `CancellationToken` to `GetConsumingEnumerable` so the loop can be stopped.
- What does a plain foreach over a BlockingCollection<T> do, without GetConsumingEnumerable?It enumerates the wrapped collection directly. For the default `ConcurrentQueue<T>` that is a moment-in-time snapshot: the loop sees the items present when it started, removes none of them, and ends without waiting for more. It is a way to inspect the collection, not to consume it.
- Add on a full bounded BlockingCollection<T> throws an exception.
- GetConsumingEnumerable ends as soon as the collection is momentarily empty.
- A plain foreach over a BlockingCollection<T> removes the items it visits.
- BlockingCollection<T> is a good fit for async methods on a web server.