A pipeline converts between one row per measurement and one column per measure around almost every step — what is wrong with that?

## What the shape is telling you A chained pipeline of steps that converts before and after almost every step has a specific diagnosis: **no stretch of it has a layout.** Each conversion exists to satisfy the step immediately in front of it, which means the arrangement is being decided locally, over and over, by whoever wrote each step. The dataset never settles into an arrangement long enough for a reader to say what form the work is in. That is a design observation, and it holds regardless of what the conversions cost. ## The cost argument, stated honestly It is tempting to say *each conversion is a full copy, so converting twice costs twice as much.* Be careful, because the cost genuinely depends on the design in front of you: - Under **eager evaluation** — where every step materialises its own result — each conversion really does produce a new table, and this is most of this class of tool. It is the reason the boundary discipline exists. - A rearrangement that only **moves a label** from one direction of the rectangle to the other, where the values already sit in the order the result wants, can be metadata work with no copy of the values at all. - Under **deferred evaluation** — where the tool builds a plan and does the work only when a result is actually asked for — an intermediate arrangement may be fused into its neighbours or skipped entirely. So do not lead with arithmetic you cannot verify. Lead with the structural point, and add the cost point in the form that survives all three designs: **a widening produces a rectangle whose extent is distinct row keys against distinct headers rather than the input's row count, and a fold back repeats every identifying column once per measure.** Neither result has the size of its input, which is why treating a conversion as free is a habit rather than a measurement. ## The repair: group, then convert once The discipline is short: 1. **Walk the pipeline and label each step with the arrangement it reads** — a condition over every measurement or an addition of a further measure reads the long layout, one row per measurement; anything taking two measures as operands, or anything shaped like a numeric block, reads the wide layout, one column per measure. 2. **Look for the runs.** Real pipelines produce two or three runs, not twenty alternations. Cleaning, filtering and accumulating cluster on one side; the analysis or the numeric step clusters on the other. 3. **Put one conversion at each boundary between runs**, and write down in the code which operations that conversion is serving. A conversion with a stated purpose survives the next edit; an unexplained one gets copied. The result is usually one or two conversions where there were eight, and — more importantly — a pipeline whose arrangement a reader can name at every point. ## Why the boundary is also where the thinking goes A boundary conversion is the only place in the pipeline where the arrangement changes, which makes it the natural place to put everything else that has to be true at the transition: what the row key of the result is, which columns were expected to come out the other side, and what the next stretch assumes about them. Spread across eight conversions, none of that can be stated anywhere. Concentrated at one, it can. It also makes the pipeline portable in a way the per-step version is not. Tools in this family disagree about which arrangement they assume, so a per-step pipeline encodes one tool's assumptions at twenty points. A pipeline with two boundaries encodes them at two, and moving the work to a tool that assumes the other arrangement means moving two conversions. ## The failure mode on the other side The opposite mistake exists and is rarer: refusing to convert at all, and writing a per-measure branch or a self-match by hand because the arrangement in play cannot express the operation directly. That is the same defect wearing different clothes — the work is still being bent around an arrangement nobody chose. The point of the discipline is not *fewer conversions* as a number; it is that each one is a decision with a reason attached. ## What an interviewer is listening for A strong answer names the structural problem first, groups the steps by what they read, puts the conversion at the boundary, and is careful about the cost claim — saying that under eager evaluation each conversion is a materialised result, while some rearrangements are metadata only and a deferred plan may collapse an intermediate. A weak answer asserts a fixed multiplier and stops there.