In a piece-by-piece read, one piece types a column as a number and a later piece as text: why, and what prevents it?
answer
- each piece decides for itself
- the guess comes from a sample
- pieces disagree at the join
- one declaration handed to every turn
basics
~20 sA reader that guesses decides per turn, and each turn sees only its own rows, so a late awkward value or a blank field flips the decision. Hand every turn the same type declaration up front.
solid answer
~50 sA shape that carries no types leaves the reader to decide, and a reader that decides does it per turn, inspecting only the rows in that turn. The first piece sees digits and returns a number; a later piece contains one value with a letter in it, or a trailing summary line, and returns text. A blank field can do it on its own: where a design marks absence with a borrowed floating-point sentinel, one blank widens a narrow whole-number column, while a design carrying a separate validity bit keeps the width. Nothing surfaces at the read. It surfaces when the pieces are combined or written into one destination, where the representations have to agree. The fix is one declaration of the types handed to every turn; where you cannot yet declare a column, declare it as characters and convert once, afterwards, on purpose.
go deeper
Know that a text shape carries no types, so something must decide them, and that you can state the types yourself instead of letting the reader guess at them.
Explain that a guessing reader decides per turn from whatever rows that turn contains, so two turns over one file can disagree, and that one declaration handed to every turn removes it outright.
Say where the disagreement surfaces — at the combination or the write, never at the read — and choose deliberately between declaring, reading as characters and converting once, and moving the producer to a self-describing shape.
The call is whether guessing is allowed in a pipeline at all. A declaration is friction once; an inference that varies with the data is a defect that arrives on someone else's schedule, in someone else's report.
## Why a type has to be decided at all Re-parsed delimited text — one record per line, every value stored as characters — carries no type information. The characters `00471` are just five characters; whether the column they sit in becomes a number or stays text is a decision somebody must make. There are only three candidates: the file, which in this shape cannot; you, by handing the reader a declaration of the types before it starts; or the reader, by inspecting some rows and guessing. What the reader inspects varies by design, and that variation is the whole of this question: - some designs fix a type from a bounded prefix of what they are about to read and never revisit it; - some scan the entire column before deciding, at the cost of an extra pass over the input; - some decide afresh for each piece of a pass in pieces; - some refuse to guess and leave every column as characters until you declare otherwise. Do not carry a specific number of inspected rows around as a fact. Establish which of those rules the reader in front of you follows, because the answer to this question is different for each. ## The loop multiplies the decision In a one-shot read the guess is made once, and if it is wrong it is wrong consistently. A pass in pieces makes the same guess once per turn, over a different stretch of rows each time. Forty turns is forty independent decisions. So the first piece, containing nothing but digits in that column, comes back as a number. Piece thirty-one contains one identifier with a letter prefix, or a line the producer appended as a summary, and comes back as characters. From the reader's point of view nothing has gone wrong: you asked a question forty times and supplied different input each time, so you got different answers. The disagreement is a property of the loop, not a bug in the reader. A second, quieter cause is absence. Where a design marks an absent value by borrowing the floating-point not-a-number sentinel, a narrow whole-number column that contains a single blank field cannot stay a whole-number column and is widened on the way in. Where a design carries a separate validity bit alongside each value, the hole is marked without touching the representation and the width is unchanged. On the first kind of design, piece twelve differs from piece eleven purely because one of its rows had an empty field. ## Where the damage shows up Not at the read. It shows up at one of three later moments: 1. **At the combination.** Stacking pieces with different representations for one column forces a single representation on the result, and the one that loses nothing is usually a character representation. Every downstream assumption about that column is now wrong, and nothing raised. 2. **At the write.** If each turn is written to one destination in a typed shape, the turn whose types disagree with what has already been written either fails or silently widens the declaration. 3. **Between partial results.** A per-turn result computed over a numeric column and another computed over a character column are not comparable, even though they came from one pass over one file. ## What to do about it 1. **Declare the types once and hand the same declaration to every turn.** This removes the entire class, costs one statement, and is the only measure that removes rather than reduces. 2. **Where you genuinely do not know a column yet, declare it as characters** for the whole pass and convert once, deliberately, afterwards. Characters are the representation that loses nothing, so you have deferred the decision instead of making it forty times by accident. 3. **Prefer a shape that declares its own types** if you can influence the producer. Then nothing is guessed, per turn or otherwise. 4. **If the reader can only guess, make it guess once** over the whole input, where that is offered, rather than afresh per turn. ## What each measure actually buys | measure | what it costs | what it removes | |---|---|---| | declare the types up front | knowing the columns; one statement | the guess, and therefore the disagreement | | read as characters, convert later | one deliberate conversion step | the disagreement; the decision is deferred, not avoided | | enlarge the rows the reader inspects | an extra pass, or memory | some of the risk; a later awkward value still flips it | | move to a self-describing shape | a change at the producer | the guess entirely, for every consumer | The row worth arguing with is the third. Enlarging the sample feels like a fix and is not one: it lowers the probability of a surprise without changing the mechanism, and it fails on precisely the file that matters, the one where the awkward value sits near the end.
- You cannot declare a column because you do not yet know what it holds. Now what?Declare it as characters for the whole pass and convert once, deliberately, afterwards. Characters lose nothing, so the decision is deferred rather than made once per turn by accident, and the conversion is then one column with one rule you chose.
- Does enlarging the rows the reader inspects fix the disagreement?It lowers the chance without removing it. An awkward value beyond whatever the reader looked at still flips the decision, and a larger sample costs an extra pass or memory. It is a mitigation; a declaration is a fix.
- Why does this problem largely disappear on a self-describing binary columnar file?Because the types are written into the file and read out of it, so the reader infers nothing and every turn gets the same declaration by construction. The inference question only exists where the shape carries no types.
saying these in an interview costs you the question
- Says the reader knows the column types, so two pieces cannot disagree.
- Thinks a larger inspection sample removes the disagreement rather than reducing it.
- Blames the combination step instead of the per-turn guess that caused it.
- Assumes a blank field costs nothing, whatever the design's absence mechanism is.
- Converts after combining, once the original characters have already been lost.