A nightly read warns on 300 fields that contradict the type it chose, then hands back a table anyway. What are the reader's options, and which do you want?

## The three dispositions, plus one When a value in the file does not fit the representation a column has been given — whether the reader guessed that representation or you stated it — the reader must do something. There are three real answers and a fourth path that is easy to overlook. | Disposition | What the table gets | What it costs | |---|---|---| | **Reject** | Nothing; the read fails at the first contradiction | A failed job, which is the outcome you are paying for | | **Coerce** | The value forced into the representation, or an absent marker where it will not go | The original value, silently, at most a warning | | **Widen** | The whole column in a text representation | The type, not the values; every later operation now behaves as text | | **Drop the record** | A table short of rows, and a line in a warning stream | The record; the exit code says nothing happened | Which of these is the default is a design choice that differs across tools and sometimes across settings of one tool. "A value that contradicts the type fails the read" is true of strict readers and false of the rest — and the rest are the majority of what people actually run. ## Why coercion is worse than it sounds Coercion does not announce the size of what it did. A field that will not fit becomes an absent marker, and there is a design-dependent consequence you should be able to state precisely: **where a tool marks absence with a representation borrowed from the numeric type, a narrow whole-number column that gains one hole cannot stay narrow and is widened on the way in** — so the width change, not the hole, is what surprises you later. **Where the tool carries a separate validity bit alongside each value, the width is unchanged and only the validity flag moves.** Same input, two different tables, neither tool wrong. Widening to text is the gentler failure and still a real one. Nothing is lost from the file, but the column's later behaviour changes wholesale: ordering becomes lexicographic, comparison becomes string comparison, arithmetic stops being available, and the memory the column occupies has little to do with what you budgeted. ## Why the warning is the actual problem In an interactive session a warning works: it is on the screen, a person is looking at the screen, and they investigate. In an unattended nightly job every property of a warning is wrong. - It does not change the exit code, so the scheduler records success. - It goes to a stream that is captured, rotated and never read. - The job produces output, so every downstream step runs normally. - The output is *plausible* — a table of the right shape with the right column names — so no shape check catches it. - Detection therefore falls to a human noticing an odd number in a report, which happens days later and is attributed to the analysis rather than the read. This is the sense in which **a reader that warns is more dangerous than one that raises**. The one that raises costs you a failed run and a page. The one that warns costs you a week of wrong answers and the credibility of everything produced during it. ## What you actually change 1. **State the types at the read.** With a stated type there is no guess to be flexible about, and the reader has a claim from you it can test each value against. This is the single change that converts the whole class of failure from silent to loud. 2. **Choose the strict disposition explicitly.** Readers expose the setting; set it, in the code, rather than inheriting whatever the default is. Write it down next to the type statement so the two are read together. 3. **Make the step's outcome the job's outcome.** If the reader will only warn, capture the warning condition and fail the step yourself. An unattended job that cannot fail is an unattended job that cannot be trusted. 4. **Compare the types you got against the types you expected.** Even where you cannot make the read strict, asserting the resulting column representations immediately after the read turns a widened column into an error at the file boundary instead of a puzzle downstream. 5. **Separate the two questions.** "Is this feed's shape changing?" and "did tonight's read give me what I asked for?" are different. The second is yours and is answered at the read; the first is a conversation with whoever produces the file, and 300 contradicting fields on a feed that has been stable for a year is the opening line of that conversation. ## The honest trade-off Strictness has a cost and it is not zero: a pipeline that fails on the first surprising field will fail on nights when a slightly sloppy feed would have been survivable, and somebody has to be on the other end of that failure. The answer is not to soften the read; it is to make the failure cheap — a clear message naming the column and the offending value, a quarantine of the night's input for inspection, and a rerun that costs minutes. **A loud failure you can act on is strictly better than a quiet answer you cannot check**, and the price of the loud failure is paid on the nights something really did change.