A line in an otherwise healthy delimited file does not fit the table's shape: what can the reader do with it?
answer
- what happens to a line that will not fit
- three exits, ranked by what they hide
- the count catches only one of them
- padding keeps the row, loses the value
- the default varies, and has changed
basics
~20 sThree dispositions: stop the read, drop the record — in silence or into a warning nobody reads — or hand it back damaged with the missing fields padded. Only the third keeps the row count, which is why it hides best.
solid answer
~50 sA reader meeting a line it cannot split into the shape the table expects has three ways out. It can **stop**, raising on the first one, so you lose the run and keep the truth. It can **drop the record**, sometimes silently, sometimes with a message to a warning stream, sometimes into a diagnostics report you have to ask for. Or it can **hand the line back damaged**: pad the fields it did not find with the absent-value marker — whatever the tool puts in a cell with no value — or absorb the surplus ones. Which of the three is the default varies by tool and has changed between releases of the same tool, so it is worth establishing rather than assuming. A row-count comparison catches the second and misses the third, because a padded record is still a row.
go deeper
Know that a line the reader cannot parse does not necessarily stop anything, and that the table you got back may be short, or the right length with wrong values in it, without a word being said.
Name the three dispositions and what each leaves you holding, and state that which one is the default is a property of the tool and its version rather than of the subject itself.
Show that you pick the disposition deliberately before the first production run, and pair it with the check that catches its particular failure: a count comparison for a drop, a look at the values for a pad.
The call is what a wrong number costs against what a stopped run costs, and whether a recurring malformed line is something a reader should absorb indefinitely or something the producing side should stop sending.
## What "does not fit" means here A **reader** is the call that turns a file into a table in one step. Working over delimited text — one record per line, every value stored as characters — it splits each line into fields and expects a fixed number of them, taken from the first line or from types you declared up front. A line is malformed when that split does not produce the expected number: one field too many, several too few. Ordinary things cause it. A separator appears inside a field's own text. A record was written by a producer that omits trailing empty fields. A file was assembled from two extracts whose shapes differ by a column. The interesting question is not why the line is broken. It is **what the reader hands you afterwards**, because that decides whether you ever find out. ## The three dispositions | Disposition | What comes back | What it costs you | How you find out | |---|---|---|---| | **Stop** | no table at all | the run | immediately, and loudly | | **Drop the record** | a table that is short | the records themselves | only by comparing counts | | **Hand it back damaged** | a table of the right length with wrong values in it | the values | only by looking at values | The ordering of danger is the reverse of the ordering of convenience. Stopping is the most inconvenient and the most honest. Padding is the most convenient and the most likely to end up in a report. ## Why the announcement is not the safeguard Most readers that continue will say something. That announcement is a courtesy, not a control, for four reasons: - **Where it goes varies.** A warning stream, a diagnostics report you must ask for, a return value nobody inspects, or nowhere. - **Scheduled runs discard it.** A job that runs nightly and succeeds writes its warnings into a place that is read only after something else has already gone wrong. - **It does not fail anything.** The run is green. Every downstream step treats the short table as the truth. - **Volume defeats it either way.** One message per bad line drowns in a file with thousands; one message per read is easy to miss entirely. So the question "would I have noticed?" has a reliable answer, and it is no. ## Choosing a disposition deliberately 1. **Start by stopping.** While you still do not know what the file contains, a raise on the first line that does not fit is information, delivered at the cheapest possible moment. You look at the offending lines, and now you know something. 2. **Then decide, with the lines in front of you.** Malformed lines are usually one recurring thing, not random damage. Once you know what that thing is, you can choose to accept it, reject it, or fix the shape you declared. 3. **If you must continue, prefer dropping to padding.** A drop moves the row count and a count comparison catches it. A pad moves nothing and a count comparison passes. Given a choice between a failure your cheapest check detects and one it cannot, take the first. ## The blind spot, and what closes it A row-count comparison immediately after the read is the standard defence, and against padding it does nothing at all: the record is still a row. What changed is that some of its fields now hold the absent-value marker, or hold values that belong in the next column along. The cheapest thing that catches it is to count absent entries per column in the same breath as the rows, and notice when a column that was full yesterday is not full today. ## What genuinely varies between tools This is the part candidates most often state as universal, and it is the part that is not: - **The default disposition differs between tools**, and has changed across releases of individual tools. - **Some tools require you to state it**, refusing to choose on your behalf. - **What "hand it back damaged" means differs.** Some pad short lines with the absent-value marker; some truncate surplus fields; some push the surplus into the last column as text. - **Where the announcement lands differs**, including the case where there is none. The practical consequence is that you cannot answer this question about your own pipeline from memory. You answer it in a minute by feeding the reader a small file you have deliberately broken — one line with a surplus separator, one line with a field missing — and looking at what comes back. That experiment survives the next upgrade, which an assumption does not.
- Which of the three dispositions does a row-count check fail to catch?Handing the line back damaged. The record is still a row, so the count is right; what changed is that some fields now hold the absent-value marker or values belonging to the next column. Catching it needs a look at the values, for instance counting absent entries per column immediately after the read and noticing a column that was full yesterday.
- Why might stopping the read be the right default even for a nightly job?Because a failed run is a known unknown that someone will look at, while a short or padded table is an unknown unknown that becomes a published number. Stopping is also recoverable: you inspect the offending lines, decide what they actually are, and re-run with the disposition you genuinely want.
- How do you find out what your reader's default disposition actually is?Feed it a small file you have deliberately broken — one line carrying a surplus separator, one line missing a field — and look at what comes back: an error, a short table, or a full table with absent markers in it. It takes a minute and it stays true across an upgrade in a way an assumption does not.
Three ways a sorting office can handle an envelope whose address it cannot read: return it to the sender, quietly bin it, or deliver it to the nearest address that looks close enough. The third is the one that produces a confident wrong outcome nobody investigates, and it is the one that leaves the day's delivery count looking exactly right.
saying these in an interview costs you the question
- Says readers warn and carry on by default
- Assumes a full-length table means a faithful read
- Treats a warning stream as a reliable control
- Thinks padding a short line is the safe option
- Cannot say which disposition a row count misses
- Believes stopping the read is always unacceptable