In the pipes-and-filters integration pattern, what makes a processing step a good 'filter,' and what do you give up by chaining many small filters instead of one larger processing step?

## The two pieces Pipes-and-filters is an architectural pattern for building a complex processing pipeline out of a sequence of independent, composable steps. - A **pipe** is the connector — a channel that carries messages from one step to the next, and typically buffers messages so producer and consumer steps can run at different speeds. - A **filter** is a processing unit that consumes a message from its input pipe, performs one well-defined transformation, validation, enrichment, or routing decision, and produces a (possibly modified) message onto its output pipe. The pattern's power comes from a **strict discipline**: a filter should not know anything about what produced the message it received or what will consume the message it emits — it only knows the shape of its input and the shape of its output. That discipline is what lets you assemble different pipelines from the same set of filters, insert a new filter into an existing pipeline without touching the filters on either side of it, and reorder or replace individual steps in isolation. ## The roles a filter plays Mechanically, filters typically fall into a few recognizable roles inside a pipeline: - a **validating filter** checks a message against rules and either passes it through unchanged or routes it to an error path; - a **translating/enriching filter** changes the message's shape or adds data (looking up a customer's tier and stamping it onto an order message, for instance); - a **routing filter** inspects the message and decides which of several downstream pipes it should go to next, rather than transforming its content. A pipeline is simply an ordered composition of these: message enters the first pipe, each filter processes and re-emits, until the last filter's output represents the fully processed result. Because each filter only ever talks to its adjacent pipes, the whole chain can be visualized and reasoned about as a straight-line (or branching) flow diagram, which is a large part of why the pattern is attractive for integration: a complex, multi-step business process — validate, enrich, deduplicate, transform format, route — becomes a legible sequence instead of one large, opaque function. ## Why decompose at all This pattern exists because monolithic processing steps are hard to reuse, hard to test in isolation, and hard to change safely: if validation, enrichment, and format translation are all one function, adding a new enrichment step means modifying that function and re-testing everything downstream of the change, and you can't reuse just the validation logic in a different pipeline without extracting it first. By forcing each unit of work to be a standalone filter with a fixed input/output contract, the pattern gets you: 1. **composability** (mix and match filters into new pipelines), 2. **independent replaceability** (swap out one filter's implementation without touching neighbors), 3. **independent scalability** (a slow filter can run more instances, or run on different hardware, without changing the others). ## The trade-off The trade-off is **granularity versus overhead**. - **Decomposing a job into many small, single-purpose filters** maximizes reuse and testability, but each filter boundary that's implemented as an actual message hop (rather than an in-process function call) adds real latency — serialization, channel transit time, and the filter's own processing time — and each filter is a separate operational unit: its own deployment, its own scaling policy, its own failure and retry behavior to monitor. - **Coarser filters** reduce that per-hop overhead and operational surface but sacrifice the composability that made the pattern attractive in the first place, since a large multi-purpose filter is exactly the monolithic step the pattern is trying to avoid. Where to draw the line — how fine-grained a filter should be — is a judgment call that trades pipeline flexibility against the number of moving parts you have to run and observe. ## Failure modes Failure modes center on partial-pipeline failure, because pipes-and-filters, unlike a single in-process transaction, has no built-in notion of 'the whole chain either fully succeeds or fully rolls back.' 1. **Partial-pipeline failure.** A message can be successfully processed by the first three filters and fail at the fourth, and by that point the first three filters' effects (if any were not purely transformational — say, a filter that also wrote an audit log or called an external system) have already happened and can't be automatically undone. This is exactly why pipelines handling business transactions rather than pure data transformation often need an explicit compensating mechanism (a saga) layered on top, rather than relying on pipes-and-filters alone to guarantee end-to-end consistency. 2. A second common failure mode is a slow filter becoming a **backpressure bottleneck**: if pipe buffering is unbounded, a slow downstream filter causes upstream buffers to grow without limit; if it's bounded, upstream filters start blocking or dropping, and either way the pipeline's overall throughput degrades to that of its slowest filter, a fact that's often invisible until load testing. ## Putting it together A concrete scenario: an inbound order-processing pipeline has four filters wired by three pipes: - a `ValidateOrder` filter (rejects malformed orders to an error pipe), - an `EnrichCustomerTier` filter (looks up and stamps loyalty tier), - a `NormalizeCurrency` filter (converts all amounts to the company's base currency), - and a `RouteByRegion` filter (sends the fully processed order to one of several regional fulfillment pipes based on shipping address). Each filter was written, tested, and can be redeployed independently; a new `FraudCheck` filter was added between validation and enrichment months later without any change to the other three.