In a .NET TPL Dataflow log pipeline of TransformBlock, BatchBlock and ActionBlock, why does memory grow without limit, and why does awaiting the last block's Completion hang?
answer
- every block buffers by default
- BoundedCapacity plus SendAsync
- Post returns false when full
- PropagateCompletion is off by default
- someone must complete the head
basics
~20 sDataflow blocks buffer without limit by default, so a fast parser fills memory ahead of a slow writer; set BoundedCapacity and feed with SendAsync. The hang comes from LinkTo without PropagateCompletion, or from a head block nobody calls Complete() on.
solid answer
~40 sEvery Dataflow block has internal buffers, and `BoundedCapacity` defaults to `DataflowBlockOptions.Unbounded` (-1), so when parsing outruns the database writer, records pile up inside the blocks. Setting `BoundedCapacity` makes a full block postpone offers, which pushes back up the chain, and producing with `await block.SendAsync(item)` waits for space; `Post` instead returns `false` and the record is lost if the result is ignored. Completion is a separate signal: `LinkTo` forwards it only with `new DataflowLinkOptions { PropagateCompletion = true }`, so after `parse.Complete()` the downstream blocks never learn the input ended and `await write.Completion` waits forever. A `TransformBlock` whose output nobody takes never completes either. With propagation on, `BatchBlock` flushes its final partial batch on completion, and a fault travels down the chain as an `AggregateException`.
code
csharp · 37 linesusing System.Threading.Tasks.Dataflow;
var bounded = new ExecutionDataflowBlockOptions { BoundedCapacity = 1000, MaxDegreeOfParallelism = 4 };
var parse = new TransformBlock<string, LogRecord>(line => LogRecord.Parse(line), bounded);
var batch = new BatchBlock<LogRecord>(500, new GroupingDataflowBlockOptions { BoundedCapacity = 1000 });
var writes = 0;
var write = new ActionBlock<LogRecord[]>(async rows =>
{
await Task.Delay(5); // stands in for a bulk insert
Interlocked.Add(ref writes, rows.Length);
}, new ExecutionDataflowBlockOptions { BoundedCapacity = 2 });
var link = new DataflowLinkOptions { PropagateCompletion = true };
parse.LinkTo(batch, link);
batch.LinkTo(write, link);
for (int i = 0; i < 10_250; i++)
await parse.SendAsync($"{i}|INFO|payment accepted"); // waits while parse is full
parse.Complete();
await write.Completion;
Console.WriteLine($"written={writes}"); // 10250: the last, partial batch of 250 is flushed
var tiny = new ActionBlock<int>(_ => Thread.Sleep(1), new ExecutionDataflowBlockOptions { BoundedCapacity = 1 });
int accepted = Enumerable.Range(0, 100).Count(i => tiny.Post(i));
Console.WriteLine($"Post accepted {accepted} of 100"); // most are declined: Post returns false when full
tiny.Complete();
await tiny.Completion;
record LogRecord(int Id, string Level, string Message)
{
public static LogRecord Parse(string line)
{
string[] p = line.Split('|');
return new LogRecord(int.Parse(p[0]), p[1], p[2]);
}
}go deeper
Know what a Dataflow block is, that blocks are linked with LinkTo, and that the producer calls Complete() on the first block when input ends.
Explain the defaults: unbounded buffers, one message at a time per execution block, ordered output, and completion that does not cross a link unless PropagateCompletion is set.
Diagnose a pipeline that eats memory or never finishes: missing BoundedCapacity, Post results ignored, links without PropagateCompletion, unconsumed outputs, and faults that never reach the tail.
Treat an in-process pipeline's buffers as a memory budget: bound every block, decide where back-pressure should stop, and decide what happens to in-flight records when a block faults.
## The pipeline and its blocks **TPL Dataflow** (`System.Threading.Tasks.Dataflow`, part of the .NET shared framework, so a .NET 10 app needs no package reference) builds pipelines out of **blocks** that pass messages to each other. A log-ingestion service typically chains three: - **`TransformBlock<TInput, TOutput>`** parses each raw line into a record. - **`BatchBlock<T>`** groups records into arrays of, say, 500. - **`ActionBlock<T>`** writes each array to the database. `LinkTo` connects a source block to a target block, and the producer feeds the head with `Post` or `SendAsync`. Each block runs its delegate on tasks from its scheduler, the thread pool by default. ## Why memory grows: unbounded buffers Each block has an **input buffer** and, for transform and batch blocks, an **output buffer**. Their defaults, confirmed by reading the options in a .NET 10 run: | Option | Default | Meaning | |---|---|---| | `BoundedCapacity` | `-1` (`DataflowBlockOptions.Unbounded`) | the block accepts every message offered | | `MaxDegreeOfParallelism` (execution blocks) | `1` | one message processed at a time per block | | `EnsureOrdered` | `true` | outputs leave in input order even when processed in parallel | | `PropagateCompletion` (on `DataflowLinkOptions`) | `false` | completion is not forwarded across a link | With unbounded buffers there is **no back-pressure**. If parsing produces 50,000 records a second and the writer stores 10,000, the difference accumulates in the parse block's output buffer and the batch block until the process runs out of memory. Setting `BoundedCapacity` changes that. A full block **postpones** offered messages instead of accepting them, the source keeps them, and when the source's own capacity is reached it postpones its upstream in turn, so back-pressure travels to the producer. Two details: - Bounding counts messages being processed as well as queued ones. - A `BatchBlock` with a bounded capacity smaller than its batch size throws `ArgumentOutOfRangeException` at construction; the capacity must hold at least one full batch. ## Feeding a bounded block: `SendAsync` versus `Post` - **`await target.SendAsync(item)`** waits asynchronously until the block accepts the message (or permanently declines it, returning `false`). This is how the producer feels the back-pressure. - **`target.Post(item)`** never waits. It returns `true` if the block accepted the message right away and `false` otherwise. Against a full block it returns `false`, so code that ignores the return value **silently drops records**. In the snippet, posting 100 items to a block with `BoundedCapacity = 1` accepted just one of them on that run. ## Why `Completion` hangs Each block exposes a `Completion` task that finishes when the block has been told no more input is coming **and** has processed and handed on everything it holds. Common reasons it never finishes: 1. **Nobody completed the head.** The producer must call `head.Complete()` when the input ends. 2. **Links without `PropagateCompletion`.** `parse.Complete()` completes `parse`, but a plain `LinkTo` does not pass that on, so the batch and write blocks wait for input forever. The second snippet shows `parse` reaching `RanToCompletion` while the target did not complete within a second. 3. **Output nobody consumes.** A `TransformBlock` cannot complete while items sit in its output buffer; an unlinked block in the snippet stayed incomplete with `OutputCount=1`. With propagation on, the end of input flows block to block, and `BatchBlock` emits its final, partial batch when it completes: the first snippet wrote all 10,250 records, the last 250 in a short batch. ## Faults An exception thrown by a block's delegate **faults that block**: it stops accepting messages (`Post` then returns `false`) and its `Completion` faults. With `PropagateCompletion = true` the downstream blocks fault too; awaiting the last block's `Completion` in the snippet threw an `AggregateException` whose inner exception was the parser's `FormatException`. Without propagation, the tail would wait forever while the head lay dead, so either propagate completion or watch every block's `Completion`. ## Parallelism and order `MaxDegreeOfParallelism` is set **per block**: raising it on the parse block parallelises parsing only, and the writer stays at one batch at a time unless you raise it there too. With `EnsureOrdered` left at `true`, a parallel `TransformBlock` still releases outputs in input order, holding back items that finish early; set it to `false` when order does not matter and you prefer lower latency.
- Why does new BatchBlock<LogRecord>(500, new GroupingDataflowBlockOptions { BoundedCapacity = 100 }) throw?A bounded `BatchBlock` must be able to hold at least one full batch; with a non-default `BoundedCapacity` smaller than `batchSize`, the constructor throws `ArgumentOutOfRangeException`. Give it a capacity of at least the batch size, often a small multiple of it so the next batch can fill while one is being written.
- Why do a parallel TransformBlock's outputs stay in input order, and when would you turn that off?`EnsureOrdered` defaults to `true`, so with `MaxDegreeOfParallelism` above 1 the block reorders results and releases each only after all earlier inputs have produced theirs. One slow record then holds back faster ones behind it. When downstream does not care about order, setting `EnsureOrdered = false` lets each result through as soon as it is ready.
- Dataflow blocks apply back-pressure by default
- LinkTo forwards completion to the target automatically
- Post waits until a bounded block has room
- MaxDegreeOfParallelism on the head block parallelises the whole pipeline
- A TransformBlock completes even if nobody reads its output