skip to content

In C#, what may a custom `Pairwise` operator over an endless `IEnumerable<Reading>` hold between elements, and what would make it hang before `Take(5)`?

level: middleimportance: should knowfreq 45%

answer

  1. bounded state per element
  2. previous item plus a flag
  3. operators that must see the end
  4. Reverse copies on first MoveNext
  5. order of Take in the chain

basics

~20 s

A streaming Pairwise keeps only the previous element and a flag, and is safe on an endless source. It hangs if it, or a stage before Take, must read to the end first, as ToList, ToArray, Count, Max, Reverse or OrderBy do.

solid answer

~40 s

A streaming operator keeps **bounded state**: `Pairwise` needs only the previous element and a `hasPrevious` flag, yields `(previous, item)` for each new reading, and never looks further ahead than one `MoveNext`. That makes it safe on an endless sensor stream, because `Take(5)` stops pulling after five pairs and disposes the chain. It hangs when something before `Take` must see the end of the source: an internal `ToList()` or `ToArray()`, a `Count()` or `Max()`, or a nonstreaming operator such as `Reverse` or `OrderBy`. In .NET's source, `Reverse`'s iterator calls `ToArray()` on its first `MoveNext`, so `Endless().Reverse().Take(1)` never yields, while `Endless().Take(1).Reverse()` returns at once. When you write an operator, decide which kind it is and document it.

code

csharp · 31 lines
csharp
foreach (var (prev, next) in Sensor.Stream().Pairwise().Take(3))
    Console.WriteLine($"#{prev.Id}->#{next.Id}: {next.Celsius - prev.Celsius:+0;-0;0}");

record Reading(int Id, int Celsius);

static class Sensor
{
    public static IEnumerable<Reading> Stream()
    {
        int[] temps = [21, 24, 24, 19];
        for (int id = 0; ; id++)
            yield return new Reading(id, temps[id % temps.Length]);
    }
}

static class SensorOps
{
    // Streams: holds one previous element, never the whole source.
    public static IEnumerable<(T Previous, T Current)> Pairwise<T>(this IEnumerable<T> source)
    {
        bool hasPrevious = false;
        T previous = default!;
        foreach (T item in source)
        {
            if (hasPrevious)
                yield return (previous, item);
            previous = item;
            hasPrevious = true;
        }
    }
}

go deeper

for a junior

Remember that streaming means one element in, results out, with no list of everything seen so far, and that Take is what stops an endless source.

for a middle

Explain bounded versus unbounded state, show Pairwise holding one element and a flag, and name the calls that force a read to the end.

for a senior

Spot a hidden ToList, Max or Reverse inside a custom operator during review, and explain why Take placed after it cannot rescue the pipeline.

for a principal

Ask that every shared operator documents its buffering behaviour and is tested against an endless source, so nobody discovers a hidden buffer in production.

## Two kinds of deferred operator The .NET documentation splits deferred LINQ operators into **streaming** and **nonstreaming**. A streaming operator works on each source element as it is read and yields as soon as it can; a nonstreaming one must read all the source data into a data structure before it yields its first result. The split matters just as much for operators you write yourself, because nothing in the type system shows it: both kinds return `IEnumerable<T>` and both are lazy until enumerated. - **Streaming:** the work per element is done with a fixed amount of state. `Pairwise`, a filter, a projection and a running total all fit. - **Bounded buffering:** a sliding window of the last `n` readings holds at most `n` elements. It still works on an endless source, because its memory does not grow with the length. - **Unbounded buffering:** sorting, reversing, grouping or "scale to the maximum" cannot produce anything until the source ends. ## Writing Pairwise as a streaming operator ```csharp public static IEnumerable<(T Previous, T Current)> Pairwise<T>(this IEnumerable<T> source) { bool hasPrevious = false; T previous = default!; foreach (T item in source) { if (hasPrevious) yield return (previous, item); previous = item; hasPrevious = true; } } ``` The state the iterator object carries between `MoveNext` calls is a fixed set of fields: its bookkeeping, the source enumerator, the current `item`, `previous` and `hasPrevious`. The flag matters for generic code: comparing `previous` to `default` would misfire for a reading that happens to equal the default value, such as `0` for an `int`. Over a million readings or an endless sensor feed, memory use stays the same. ## What makes an operator hang Each of these reads the whole source before the operator yields anything: 1. Calling `source.ToList()` or `source.ToArray()` at the top "to make it easier". 2. Calling an aggregate first: `source.Count()`, `source.Max()`, `source.Last()`. 3. Delegating to a nonstreaming operator internally, such as `source.Reverse()` or `source.OrderBy(...)`. On a finite source these are only memory and latency costs. On an endless one the first `MoveNext` never comes back; a growing buffer such as `ToList` keeps reading until memory runs out. The `Reverse` case is easy to see in the runtime's `ReverseIterator<TSource>`: its `MoveNext`, on the first call, does `TSource[] buffer = _source.ToArray();` and only then starts yielding from the end. The second example on this page guards an endless iterator with a counter, and `Reverse().Take(1)` trips the guard after a million reads. ## Where Take sits decides everything `Take(n)` is what ends a pipeline over an endless source, and it is careful about it. In .NET 10, `Take` on a plain enumerable uses an iterator that checks how many elements it has handed out **before** calling the source's `MoveNext`, so after the `n`th element it never pulls another one; then it disposes the source enumerator. | Chain | What happens on an endless source | |---|---| | `Endless().Pairwise().Take(5)` | five pairs, six readings, then disposal | | `Endless().Take(6).Pairwise()` | the same five pairs | | `Endless().Reverse().Take(1)` | never yields; `Reverse` buffers forever | | `Endless().Take(1).Reverse()` | returns the single element immediately | A `Take` at the end of the chain protects only the stages that yield before the source ends, whether they stream or keep a bounded buffer. It cannot help a stage that refuses to yield until the source ends, because `Take` never receives a first element to count. ## Practical rules for operator authors - State in the XML summary whether the operator streams, buffers a bounded amount, or buffers the whole source. - Test every streaming operator against an endless source cut by `Take`; a hang in the test is the cheapest place to find a hidden buffer. - When a whole-sequence statistic is genuinely required, buffer once, on purpose, and say so; the question about enumerating a source twice covers why reading it again is worse. - Size any bounded buffer from a parameter the caller controls, such as a window size, never from the data itself, or a long run of unusual readings turns a bounded buffer into an unbounded one. - Keep per-element state in locals of the iterator. They live as fields of the generated iterator object for exactly as long as the enumeration does.

  • When does `Reverse` actually read its source?
    Not at the call. `Reverse` returns a `ReverseIterator<TSource>`; on its first `MoveNext` it copies the whole source with `ToArray()` and then yields from the end. So the cost is deferred but total, and on an endless source that first `MoveNext` never returns.
  • Why does `Take(5)` not pull a sixth element from the source?
    In .NET 10, `Take` over a plain enumerable compares the number of elements already taken with its limit before calling the source's `MoveNext`. Once the limit is reached it stops and disposes the source enumerator, so a live sensor is not asked for a reading nobody will use.
  • Is a window of the last ten readings still safe on an endless stream?
    Yes. It buffers, but the buffer is bounded by ten elements and it yields as soon as each window is full, so memory does not grow with the stream and `Take` can stop it. Only buffers that grow with the source length make an operator unsafe on an endless one.

  • Pairwise has to collect the readings into a list before pairing them
  • Take(5) at the end makes every operator before it safe on an endless source
  • Reverse reads its whole source at the moment it is called
  • Any deferred operator is automatically a streaming one
  • Comparing the previous element to default is a safe first-item check