A script adds twenty small derived fields one at a time to a wide table and each add runs slower than the last — what explains it?
answer
- cost climbs with each attached field
- work proportional to what is already there
- a slab rebuilt, or a holding copied, per add
- attach one field to a wide and a narrow table
basics
~20 sEach add is doing work proportional to the table already built, not to the new field — a shared slab rebuilt per add, or a fresh holding materialised per step. Twenty adds then cost twenty table-sized copies. Compute the fields first and attach them once.
solid answer
~50 sA cost that climbs with each attached field points at storage being reorganised per add, not at the expressions. Two designs produce it: one that packs every same-representation field into a single slab, where adding another field of that form allocates a bigger slab and copies the existing ones in; and one that materialises a whole new holding per step, copying the values it touches. Either way the work per add tracks the table already there, so twenty adds cost roughly twenty table-sized copies — the total grows with the square of the number of adds. It does not reproduce everywhere: a design that shares the untouched runs allocates only the new field, and a deferred design does nothing until a value is asked for. Confirm it, then compute all twenty fields and attach them in one step.
code
pseudocode · 10 lines# one at a time: each attach MAY reorganise storage sized by the whole table
for name, expression in twenty_derived_fields:
values = evaluate(expression, table)
table = attach_field(table, name, values) # up to 20 table-sized copies
# batched: the derived values are built first, the table is reorganised once
new_fields = {}
for name, expression in twenty_derived_fields:
new_fields[name] = evaluate(expression, table)
table = attach_fields(table, new_fields) # at most 1 table-sized copygo deeper
The signal to notice is that the cost rises with each step rather than staying flat. That points at something table-sized being redone every time, not at the work you asked for.
Explain the mechanism: a shared slab reallocated per added field, or a whole holding copied per step, either of which makes total work grow with the square of the number of adds.
Show the discriminating test — the same field attached to a narrow and a wide table — and the batched repair, and say plainly that the effect depends on the storage design rather than on tables in general.
The lever is a convention: derived fields are computed as a set and attached once. It removes a quadratic failure mode from code written by people who will never know which holding they are on.
## Reading the symptom The useful detail is not that the script is slow but that **each add is slower than the one before it**. The twenty expressions are independent of each other, so their own cost does not grow as the script proceeds. Something whose size *does* grow with each add is being touched every time — and the only thing that grows is the table's own storage. That narrows the hypothesis to one sentence: **the per-add work is proportional to the table already built, not to the field being added.** ## The two mechanisms that produce it 1. **A shared slab being rebuilt.** Where a design packs every field sharing a **column representation** into one two-dimensional slab, a new field of a form the slab already holds cannot simply sit beside it. A bigger slab is allocated and the existing fields of that form are copied in. Add the twentieth narrow field to a table that already has nineteen and you move nineteen fields' worth of values to place one. 2. **A whole holding materialised per step.** Where each step produces a new holding by copying the values it touches, the copy is table-sized whatever the step was. Twenty steps, twenty copies. In both, the per-add cost climbs as the table widens, and the total is the sum of that climb — work growing with the **square** of the number of adds, not linearly with it. ## Why it will not reproduce everywhere This is the part to say out loud, because the claim "a step that produces a new object roughly doubles peak memory" is only true of one family of designs: - **Designs that copy what they touch** pay the full table-sized cost per step, and at the moment of the swap the source and the result are both resident. - **Designs immutable by default that share untouched runs** allocate only the new field and point at the existing runs. The slowdown simply does not appear, and peak memory rises by one field, not by a table. - **Deferred designs** record the step and do nothing until a value is asked for. Nothing is built at the point the add is written, so timing the add measures nothing at all. So the diagnosis is not "tables are like this"; it is "this holding is like this", and part of the answer is knowing which one you are on. ## Confirming it before changing anything The cheapest discriminating test holds everything constant except how much table is already there: - attach **one identical field** to a table with one field, and to the same table with nineteen fields already attached; - if the second attach costs materially more, the add is touching storage that already exists, and the hypothesis is confirmed; - if both cost the same, look elsewhere — the expressions, the source of the values, or the materialisation of something the script did just before. What does **not** discriminate: halving the row count (almost everything in the script scales with rows), reordering the adds (the same total storage is touched), or watching total memory at the end (it says nothing about whether each individual add reallocated). ## The repairs, in order of preference - **Compute all the fields, then attach them in one step.** One reorganisation instead of twenty. This works under every holding and is the only change that needs no knowledge of which one you are on. - **Build the derived fields as standalone values and combine once at the end**, if the intermediate table is never read between adds. The combination is then a single table-building step. - **Give the derived fields a representation the table does not already hold**, where the consolidating design is the cause and combining is genuinely impossible — a new form gets its own slab rather than rebuilding an existing one. This is a real lever but a fragile one, because it depends on a storage detail the logical picture does not expose. - **Do the work under a deferred design** where one is available, so the twenty steps are recorded and the holding is built once. ## What not to conclude - Not that the machine is short of memory. Memory pressure produces a cliff, not a cost that climbs smoothly with each attached field. - Not that the derived expressions are at fault. They are independent, and the test above separates them cleanly. - Not that this is a universal property of tables. It is a property of a holding that reorganises on a field add, and the same script on a sharing or deferred design shows none of it. - Not that the fix is to add the fields in a different order. The total storage touched is the same either way. The transferable lesson is the one this whole subject keeps returning to: **the logical picture does not price the operation.** "Add a field" looks like the cheap direction and usually is, but the degree lives in the storage, and a loop that performs the cheap direction twenty times can cost more than the expensive direction performed once.
- How would you confirm the rebuild hypothesis without instrumenting the library?Attach one identical field twice: once to a table carrying a single field, once to the same table already carrying nineteen. If the second attach is materially slower, the add is touching storage that already exists. Both attaches do identical work on the new field, so the difference can only come from the table around it.
- Does the same reasoning apply to twenty successive record appends?The direction does, the degree does not. A holding that keeps each field as one run has to make room in every run per append; a chunked holding takes each batch as one more chunk and copies nothing. The repair is identical either way: accumulate cheaply and attach in one batch.
- The script is unchanged but a colleague cannot reproduce the slowdown. What does that tell you?Most likely that their holding shares the runs it did not touch, so each add allocates only the new field, or that it defers the work until a value is asked for. It is evidence about the storage design, not evidence that the original measurement was wrong.
saying these in an interview costs you the question
- Blames the derived expressions, which do not grow more expensive as fields accumulate
- Assumes every design reorganises storage on a field add
- Attributes it to the table growing, when twenty narrow fields barely widen it
- Concludes the machine is out of memory without checking whether each add copies
- Proposes attaching the fields in a different order as the fix