In a Pipes and Filters pipeline, what's the difference between a push-based and a pull-based data flow model, and what trade-off does each make?

## Two ways control can flow Push and pull are the two fundamental ways control can flow through a pipe connecting two filters, and the choice between them shapes almost everything about how a pipeline handles speed mismatches, memory, and failure — it's one of the first concrete design decisions you make once you move past the abstract idea of 'filters connected by pipes.' ## Who is in control - **Push.** In a push model, the producer filter is in control: as soon as it has a unit of output ready, it writes it to the pipe and moves on, without waiting for the consumer to ask for it. A clean push example is an event-driven filter that registers a callback and gets invoked by whatever upstream mechanism produces records, with no ability to say 'don't call me yet.' - **Pull.** In a pull model, the consumer filter is in control: it calls something like `next()` on its upstream source and blocks (or gets nothing back) until the upstream filter has a unit ready, only advancing when it is itself ready for more work. An iterator, a generator, and a database cursor are the classic pull abstraction — the consumer literally drives the pace of the entire chain by how often it asks for more. | | Push | Pull | |---|---|---| | **In control** | the producer filter | the consumer filter | | **Concrete form** | an event-driven filter that registers a callback | an iterator, a generator, a database cursor | | **Natural fit** | sensor readings, incoming network packets, UI click events | a batch job reading rows from a database cursor | | **Failure mode** | unbounded queue growth, OOM kills, silent data loss | a slow or blocked consumer stalls the entire chain upstream of it | ## Which one fits a source The reason this distinction exists is that real filters don't all run at the same speed, and someone has to decide what happens when they don't. - **Push** is a natural fit when the producer's timing is externally determined and can't be deferred — sensor readings, incoming network packets, UI click events — the data simply arrives when it arrives, and the pipeline's job is to keep up. - **Pull** is a natural fit when you want the consumer to set the pace and you don't want unconsumed data piling up anywhere — a batch job reading rows from a database cursor typically pulls, because there's no inherent urgency to have the next row ready before anyone asks for it. ## The trade-off The trade-off is essentially about who bears the cost of a speed mismatch and how visible that cost is. With pure push, if the consumer is slower than the producer, output has nowhere to go except an ever-growing buffer, which either grows unbounded (risking out-of-memory) or is capped, in which case you need an explicit backpressure signal — without that, you silently drop data or crash. Pull avoids this by construction: the consumer only ever asks for the next item when ready, so there's no unconsumed backlog to manage, but the cost shows up elsewhere — a producer that generates data on its own schedule can't simply be paused between pull requests, so pure pull is a poor fit for genuinely push-driven sources, and you often need an adapter (a bounded buffer) to bridge the two, which reintroduces the same backlog question in a smaller, more controlled form. Pull also complicates fan-out: if two independent downstream filters each want to pull from the same upstream filter at their own pace, you either need the upstream to support multiple independent cursors or you accept that one slow puller stalls the shared source for everyone. ## Failure modes in production In production, each model has its own failure mode. - **The push failure mode** shows up as unbounded queue growth leading to OOM kills, or as silent data loss when a bounded queue drops the oldest/newest item under pressure — this is one of the most common causes of a streaming pipeline falling over under load spikes it handled fine at steady state. - **The pull failure mode** shows up differently: a slow or blocked consumer stalls the entire chain upstream of it, which can starve unrelated consumers sharing the same source, or cause request timeouts in systems where the 'pull' is implemented as a synchronous call with a deadline. ## Where the two meet A concrete real-world instance of a hybrid approach is the Reactive Streams specification (implemented by libraries like Project Reactor, RxJava, and Akka Streams), which explicitly models a pull-with-push-notification protocol: the consumer signals how many items it's willing to receive via a `request(n)` call, and the producer pushes up to that many items and then waits — a demand-signaling design created specifically because pure push causes unbounded buffering and pure pull can't handle sources that genuinely produce data on their own schedule, like network sockets.