Pipes and Filters shows up in ETL jobs, streaming systems, and CLI toolchains, but it's a poor fit for some problems. What kind of problem should make you reach for a different architectural style instead?

## Where the style earns its place Pipes and Filters earns its place whenever a problem's natural shape is 'take data, apply a sequence of transformations to it, and each transformation's output feeds directly into the next.' The mechanism that makes this fit well is that the style requires almost no coordination logic beyond the wiring itself — there's no orchestrator deciding what happens next, no shared state to manage, just a chain where each filter's output is unambiguously the next filter's input. This shape shows up constantly in real systems: - **ETL pipelines** extract raw records from a source, validate their schema, transform/enrich them, and load them into a warehouse — each of those is naturally a separate concern that only needs the previous stage's output. - **Compiler construction** is a textbook academic example: lexing turns source text into tokens, parsing turns tokens into an AST, semantic analysis and optimization passes transform the AST, and code generation turns it into machine code or bytecode — each pass consumes the previous pass's output format and produces the next pass's input format. - **CLI toolchains** are the most visible everyday example — chained shell commands. - **Streaming aggregation** (windowed counting, moving averages over an event stream) is the continuous-data analog of the same shape. ## Why those problems fit The underlying reason Pipes and Filters fits these cases is that the problems themselves have no need for backward communication or shared mutable context — validating a record doesn't need to ask the extraction stage a follow-up question, a parser pass doesn't need to renegotiate anything with the lexer once tokens are produced. When a problem genuinely has that shape, forcing it into a different style usually adds complexity without adding value. ## The four mismatches The trade-off, and the reason it's a poor fit elsewhere, comes from exactly the properties that make it good at what it's good at: strict one-directional flow and stage independence. | Mismatch | The better fit | |---|---| | **dynamic, business-rule-driven routing** | a workflow/orchestration engine | | **bidirectional interaction** | a synchronous call or a different collaboration style | | **transactional integrity across stages** | a single service's database transaction, or an explicit saga with compensating actions | | **granularity mismatch** | plain in-process function composition | 1. **Dynamic, business-rule-driven routing.** The first mismatch is dynamic, business-rule-driven routing. If 'what happens to this record next' depends on evaluating conditions that can send it down genuinely different paths, retry it, hold it for manual review, or route it based on data discovered mid-pipeline, a fixed linear chain of filters becomes an awkward place to encode that logic — you end up smuggling conditional/branching logic inside filters that were supposed to be simple, single-purpose transforms, or building an ad hoc mini-orchestrator on top of the pipeline anyway. In that situation, an actual workflow/orchestration engine that's designed to express branching, retries, and compensation logic as first-class citizens is a better fit, because it gives you visibility and control over the routing decisions instead of hiding them inside filter internals. 2. **Bidirectional interaction.** The second mismatch is bidirectional interaction. Pipes and Filters models one-way data flow; if stage B needs to ask stage A a question and get an answer back before proceeding — not just consume A's already-produced output — that's a request/response relationship, which calls for a synchronous call or a different collaboration style, not a pipe. 3. **Transactional integrity across stages.** The third mismatch is transactional integrity across stages. Because filters are independent and typically communicate through pipes with no shared transaction context, there's no natural single place to say 'either all of these stages' effects happen, or none do.' If a business operation truly needs atomicity across what look like sequential steps — e.g. 'debit account A and credit account B must both happen or neither happens' — you want that inside a single service's database transaction (or an explicit saga with compensating actions), not spread across independent pipeline filters where a mid-pipeline crash leaves partial, hard-to-reconcile effects. 4. **Granularity mismatch.** The fourth mismatch is granularity mismatch: if each 'record' flowing through the pipeline is tiny and the actual transformation logic per record is trivial, the fixed cost of crossing a pipe boundary — serializing, writing, reading, deserializing, and the scheduling/context-switch overhead if filters run as separate processes or threads — can dwarf the cost of the transformation itself. In that case, plain in-process function composition beats a 'real' pipe-and-filter decomposition on both latency and simplicity, and you should keep the pipeline concept at a coarser granularity (batches of records, not individual fields) or drop it in favor of ordinary function calls. ## Where the boundary shows up A concrete real-world instance of hitting this boundary is a payment-processing flow: the parts that are genuinely a linear transform-and-enrich sequence (parse the incoming transaction message, validate its format, enrich it with merchant metadata) fit Pipes and Filters well, but the actual 'authorize, reserve funds, and settle' step needs transactional guarantees and often conditional retries/compensation on failure — which is exactly why real payment systems layer a saga or orchestrator on top of, rather than purely inside, a pipeline of filters.