A per-row loop over a 50-row column is rewritten as one whole-column expression and gets slower — why?
answer
- two cost terms, not one
- one term is paid per call
- checks, type resolution, allocation
- fixed cost divided by n
- crossover moves with design and chain
basics
~20 sEvery whole-column call pays fixed setup before touching a value: checking arguments, resolving types, allocating the result. Over 50 values that setup costs more than the loop it replaced; over a million it disappears into the per-value work.
solid answer
~50 sWorking column-wise — saying the operation once for the whole column rather than once per row — wins by amortising, not by magic. A call has two cost components: a **fixed part paid once** (argument checking, working out the operands' stored representations, selecting the compiled routine, allocating a result buffer, and on some designs building a small execution plan for the expression) and a **per-value part paid n times** inside compiled code. The per-value part is far cheaper than an interpreted loop iteration, but the fixed part is not free. So the comparison is roughly `fixed + n × cheap` against `n × expensive`, and below some n the fixed term dominates. At 50 rows, with each operator in a chain paying its own fixed cost, the rewrite can lose outright. The crossover is not a constant: it moves with the design, the operator and the chain length, so measure it rather than quoting a number.
go deeper
Recall that saying an operation once for a whole column is the normal habit, because it replaces many interpreted steps with one call. Also recall that normal is not the same as always.
Explain the cost as a fixed per-call part plus a cheap per-value part, then show why dividing the fixed part by a small number of values makes the rewrite lose to the loop it replaced.
Show how you would locate the crossover on your own data and tool instead of quoting one: vary the length, the chain depth and the operands' stored representation, and time the real expression.
Weigh a blanket review rule that every loop must be rewritten column-wise against case-by-case judgment, given that most columns in a codebase are long enough for the rule to be right and a few are not.
## Two costs live inside one call Working column-wise means saying an operation once for a whole column instead of once per row. What happens underneath splits into two costs that behave completely differently as the column grows. - **A fixed cost, paid once per call.** Checking the arguments; deciding what each operand's stored representation is — the single representation every value in a column shares — and whether either must be converted; selecting the compiled routine that matches; allocating a result buffer of the right length; and, on designs that build an execution plan for each expression before running it, constructing and inspecting that plan. - **A per-value cost, paid once per value inside compiled code.** The arithmetic itself, walking packed bytes with no type decision per value and no wrapping of each value as a full language object. The per-row loop being replaced has almost the opposite profile: nearly no fixed cost, and a large per-value cost, because each iteration re-decides types, unwraps and re-wraps values, and pays the host language's own interpretation overhead. | form | fixed cost | cost per value | total at n = 50 | total at n = 5,000,000 | |---|---|---|---|---| | per-row loop | negligible | expensive | 50 × expensive | 5,000,000 × expensive | | whole-column call | real | cheap | fixed + 50 × cheap | fixed + 5,000,000 × cheap | Dividing the fixed cost by n is the whole story. At five million values it is invisible. At fifty it is the bill. ## Why the fixed cost is larger than people expect - **A chain multiplies it.** Under eager evaluation, where each operator returns a complete result before the next sees it, four operators are four calls and four fixed costs — over the same fifty values. - **Conversion at the boundary.** If an operand is not already in the packed representation the routine wants, it is converted first, and that conversion is itself per-call work with a per-value component. - **Allocation is not free.** A fresh result buffer must be obtained and, the first time it is written, its pages touched. - **Label alignment, where the operands carry it.** If both operands are columns that carry row labels, the operator must first work out how the two label sets line up before any arithmetic happens. That is a set operation over labels, and at small n it can cost more than the arithmetic. Operands that are bare positional buffers skip it entirely and simply require equal lengths. ## What varies between designs The shape of the curve is general; the size of the fixed term is not. - Designs that dispatch straight into a compiled routine have a small fixed cost — argument checks and an allocation. - Designs that record the expression and plan it before running — deferred evaluation, where writing the expression only notes what is to be done and nothing runs until the answer is asked for — have a **larger** fixed cost and a better large-n story, because planning lets them fuse several operators into one pass. - Label-carrying tabular designs pay alignment; bare packed buffers do not. So a crossover measured on one tool is a fact about that tool. Reporting it as "the" crossover is the mistake this question exists to catch. ## Finding the crossover honestly 1. Time the **real expression**, not one operator, at several column lengths spanning three or four orders of magnitude. 2. Plot total time against length. The **intercept** is the fixed per-call cost of the whole expression; the **slope** is its per-value cost. 3. Read the crossover off against the loop's line rather than guessing it. A flat region at small n, where time barely responds to length, is the fixed term dominating. 4. Re-measure if the stored representation changes, if the chain gains a step, or if the tool changes, because all three move the intercept. ## What to do when n really is small - Keep the values in plain host-language form and do the arithmetic there; at a few dozen values, the machinery of a typed column is overhead with nothing to amortise it against. - **Batch.** If small work arrives repeatedly, gather many pieces into one column, issue one call, and split the result. That converts thousands of fixed costs into one and is usually a bigger win than making any single call faster. - Do not let a microbenchmark at one size become a rule. Most columns in a real codebase are long enough for the column-wise habit to be right, which is exactly why the habit exists. ## The rule this does not overturn None of this makes the per-row loop a good default. It makes "always faster" a false statement and "faster above some length, and that length is measurable" a true one. The senior half of this subject is knowing which side of that length you are on before you rewrite anything.
- Does chaining four operators over a short column pay the fixed cost once or four times?Under eager evaluation, four times — each operator is its own call, with its own argument checking, type resolution and result allocation, and each leaves a complete result behind. Under deferred evaluation the chain may be recorded and planned once, then executed as a single pass, which trades four small fixed costs for one larger planning cost.
- If the fixed cost is the problem, why not concatenate many small columns into one?That is exactly the fix where it is available. Gather the pieces into one column, run one call, then split the result: one fixed cost instead of thousands. It is why batching an interactive path usually beats optimising each individual call. It works only when the pieces share a stored representation and the operation is independent per position.
- How would you tell per-call setup apart from per-value cost in a measurement?Time the same expression at several column lengths and plot total time against length. The intercept is the fixed per-call cost and the slope is the per-value cost. A flat region at small lengths, where time barely changes as the column grows, is the fixed term dominating everything else.
Preheating an oven takes twenty minutes whether you are baking one biscuit or two hundred. For two hundred it is nothing per biscuit; for one it is the entire job, and the frying pan wins.
saying these in an interview costs you the question
- Says the whole-column form is faster at any size, without qualification.
- Quotes a universal crossover row count learned from one tool.
- Counts only the per-value work and ignores the per-call setup entirely.
- Thinks the fixed setup is paid once per value rather than once per call.
- Benchmarks once at a million rows and makes a rule for all sizes.