You stacked two pieces and got two half-empty columns where you expected one - what happened, and why no error?
answer
- two names, one meaning
- the holes were manufactured, not read
- filled shares that sum to the row count
- reconciled by name, so the union was kept
- compare the pieces' column names as sets
basics
~20 sThe column is named differently on the two pieces, so a stack that reconciles by column name treated them as two columns and took the union of the two column sets, filling each gap with the absent-value marker. Nothing was wrong from the tool's point of view, so nothing was raised.
solid answer
~50 sThe two pieces disagree on a column name - a typo, a casing difference, a trailing space, or an upstream rename - and the stack reconciled them **by name**. Since one name appeared on only one piece and the other on only the other, the result carries both, and each is filled exactly for the rows that came from its own piece and holds the tool's marker for an absent value everywhere else. That is why the tell is two columns whose filled shares add up to the total row count. No error is raised because a forgiving union is a defined, intended behaviour: it is what makes it possible to stack pieces that genuinely differ. Not every tool does this - some refuse until the column sets match, and some reconcile by position instead - so the first thing to establish is which rule yours follows.
go deeper
Recognise the pattern: two similar column names, each filled for only part of the rows. That means the pieces spelled one thing two ways, and a forgiving combine kept both. Compare the two pieces' column names before you combine them.
Explain the mechanism. A stack that reconciles by name takes the union of the two column sets and fills the gaps with the tool's absent-value marker, so a name present on one side only becomes its own partly filled column and no error is raised.
Show that the behaviour is a property of the tool, not of the operation - union and fill, refuse, or reconcile by position are all real - and that your seam declares the expected column set so an unsanctioned difference fails there rather than downstream.
The tradeoff is forgiveness against friction. A forgiving combine lets many producers evolve independently and pays for it with failures that surface as bad numbers; a strict one turns those into loud failures at the seam and makes every legitimate schema change a coordinated release.
## What you are actually looking at You combined two pieces that mean the same thing, one under the other, and the result contains two columns where your mental model had one. Each is populated for part of the rows and empty for the rest. That pattern - **two similarly named columns whose filled row sets are disjoint and together account for every row** - is close to diagnostic. It says the two pieces used two different names for the same thing, and that the operation reconciled them by name. The empty cells hold the **absent-value marker**: the tool's representation of a value that is not there. It did not come from the source. It was manufactured by the combine step out of rows that were, individually, perfectly complete. That distinction matters because every downstream consumer will read those holes as "the source had nothing here". ## Why nothing was raised The forgiving behaviour is not a bug and not an oversight. Being able to stack pieces whose column sets differ is genuinely useful: a later month gains a column, an older extract lacks one, and you would rather have the union than nothing. The operation cannot tell the difference between that legitimate case and your typo, because from its point of view both are simply a name present on one side only. So the absence of an error carries no information at all. What carries information is the shape of the result, and you have to look at it. ## The three behaviours, and which one you have Tools across this family do not agree, and a complete answer names all three rather than assuming the one you have used: | Behaviour | What happens to a name present on one piece only | The tell | |---|---|---| | Reconcile by name and fill | kept as its own column, holes for the other piece's rows | two partly filled columns; no error | | Refuse the operation | the call fails until the column sets are made to match | an error at the seam, which is the cheap outcome | | Reconcile by position | names are not consulted; the value lands under whichever heading is in that slot | no error, no holes, and values under the wrong headings | The first is the dangerous one, precisely because it is the one that looks like it worked. The third is dangerous in a different way and is worth knowing about separately. The second is the one you would choose if you were designing the seam. ## The tells, cheapest first 1. **Compare the column counts.** The combined table carrying more columns than either piece did is the first signal, and it costs nothing. 2. **Compare the two pieces' column names as sets** - what is in one and not the other, in both directions. Two names that differ only by casing, by a trailing space, or by one character are the answer. 3. **Look at the filled share of each suspicious column.** If two columns' filled shares sum to the total row count and never overlap, they are the same column under two names. 4. **Check the row count against the sum you predicted.** It will usually be right here - this failure does not lose records - and confirming that narrows the diagnosis to the columns. ## The usual causes - A **rename upstream** that reached one producer of the pieces and not the other. - **Casing**, a **trailing space**, or an invisible character in a header that was typed by hand once. - The two pieces were **built by different code paths** - one from a reader, one constructed in the process - and only one of them applied the renaming step. - A column that was **added later**, so the older pieces genuinely lack it. Here the union is the correct result and the holes are honest; you still want to have decided that on purpose rather than discovered it. ## Fixing it, and fixing the class of it The immediate fix is to normalise the names on each piece before combining - a single mapping applied to every piece, rather than a repair applied to the combined table afterwards. Repairing afterwards means merging two partly filled columns into one, which is more code and loses the record of what happened. The durable fix is to stop relying on the tool's forgiveness. State the column set you expect, compare each piece against it before the stack, and fail on a difference you did not sanction. If your tool offers a strict same-columns rule, turning it on converts this whole failure mode into an error at the seam, where a person who understands the two producers is standing, rather than into two half-empty columns discovered by whoever averages the column three steps later - and gets an average over the rows from one piece only.
- Why does this failure damage an average more than it damages a row count?Because the row count survives it. Every record is still present, so the count matches your prediction. The values, though, are split across two columns, so an average over either one is computed on the rows from a single piece while silently presenting itself as an average over everything. The denominator moved and nothing said so.
- When is taking the union of two differing column sets the right behaviour rather than a hazard?When the pieces genuinely differ - a column introduced part way through a history, or an optional attribute only some sources carry. Then the holes are honest and the union is the result you want. The hazard is not the union itself; it is getting it by accident, so the fix is to declare the expected column set rather than to distrust the operation.
- What would you check first if the combined table has the expected columns but a column is unexpectedly empty for part of the rows?Whether the emptiness lines up with a piece boundary. If exactly the rows from one piece are empty, the column was absent or differently named on that piece and something later merged the two names. If the emptiness is scattered, it is a source problem rather than a combining problem, and the combine step is exonerated.
saying these in an interview costs you the question
- Believes every tool fills the gaps rather than refusing the stack.
- Reads two half-empty columns as a source problem, not a naming one.
- Assumes a hole means the source really had no value.
- Checks only the row count, never the column set, after stacking.
- Treats the absence of an error as evidence the combine was correct.
- Repairs the combined table instead of normalising each piece first.