What makes filters in a Pipes and Filters pipeline composable — able to be freely reordered, swapped, or reused across different pipelines — and what design discipline is required to keep that property?

## Composability is earned Composability in Pipes and Filters is not free — it is the direct consequence of a specific design discipline, and it's worth being explicit about what that discipline actually requires, because violating it is the single most common way real pipelines quietly stop being composable even though they still look like a chain of independent stages. ## The data contract The mechanism starts with the data contract. Every pipe carries data in some agreed shape: - a line of text; - a CSV row; - a JSON record with a known set of fields; - a Protocol Buffer message of a given schema. A filter's entire interface to the rest of the world is 'I consume data matching contract X and I produce data matching contract Y.' As long as two filters' contracts line up, they can be connected in any order relative to other filters that also satisfy compatible contracts, without either filter's code changing. This is why `grep` can sit before or after `sort` in a Unix pipeline — both consume and produce lines of text, so the contract is satisfied regardless of position. The pipe itself does no transformation; it just relays bytes/records from one filter's output to the next filter's input, which is exactly why the contract, not the pipe, is what does the coupling (or decoupling) work. ## Common ways the discipline breaks The reason this discipline matters is that filters are tempting to write in ways that quietly break the contract-only coupling. 1. **Hidden state.** The most common violation is hidden state: a filter that accumulates data across records and depends on execution order or on a companion filter having already run (e.g. a 'deduplicate' filter that assumes an upstream 'sort' filter has already grouped duplicates adjacently). The code still looks like an independent filter — it reads input, writes output — but it is actually coupled to a specific upstream filter's behavior that isn't expressed anywhere in its declared contract. Reorder the pipeline or swap out the sort filter for a different implementation and the deduplicate filter silently starts producing wrong results, often without erroring, which is worse than a crash because it looks like everything is fine. 2. **Side effects outside the pipe.** Another common violation is filters with side effects outside the pipe — writing to a shared file, mutating a global cache, calling an external API with ordering assumptions — which couples filters through a channel invisible to the pipeline's wiring diagram. ## The trade-off The trade-off of enforcing strict composability is **upfront cost versus long-term flexibility**. Designing genuinely stateless, contract-pure filters takes more discipline than just writing whatever code gets today's job done — you have to resist the shortcut of reaching into shared state when it would be quicker, and you have to define and often version a schema rather than let each filter assume whatever shape is convenient. The payoff is that filters become genuinely reusable assets: the same 'parse CSV', 'validate schema', 'enrich with geo-IP' filter can be dropped into a dozen different pipelines. Without that discipline, you still have code organized as a chain of steps, but you've lost the actual architectural benefit — every 'filter' is really only usable in the one pipeline it was written for, which is architecture-in-name-only. ## What it looks like in production In production this shows up as a specific class of bug: - a pipeline that has worked correctly for months breaks the moment someone reorders stages, parallelizes two stages that used to run sequentially, or reuses a filter in a new context — and the root cause is almost always an **undeclared dependency** (shared mutable state, an assumed ordering, an assumed volume/batching behavior) that the contract never captured; - **schema drift** is a related failure mode: a filter starts emitting an extra field or a renamed field, and every downstream filter that structurally depended on the old shape breaks, usually with no compile-time warning since text- or JSON-based pipes carry no static typing across process boundaries. ## A concrete example A concrete real-world example is a data-engineering ETL pipeline built with discrete stages — extract-from-source, validate-schema, deduplicate, enrich-with-reference-data, load-to-warehouse — implemented as independently deployable steps, for instance in a workflow tool like Apache Airflow. Teams that keep each stage's contract explicit (a versioned Avro or Protobuf schema on the intermediate storage) can swap the 'deduplicate' stage for a smarter implementation, or insert a new 'PII-redaction' stage between extract and load, without touching the other stages' code. Teams that let stages assume undocumented facts about each other end up with a pipeline that only works in its original order and breaks in confusing, hard-to-trace ways whenever anyone tries to change it.