Where in a multi-step file is a column's representation worth asserting, and what must the assertion claim?
answer
- two different claims, one column
- held as, not convertible to
- assert at boundaries, not everywhere
- the declared reading is bookkeeping
basics
~20 sAssert at each boundary where the representation can change and just before the first step whose correctness depends on it. The claim must be that the column is held as a number, not merely that its values could be read as one.
solid answer
~50 sTwo claims get confused here, and only one of them catches this defect. "Every value in this column can be read as a number" is satisfied by a column of digits held as text — which still orders and compares character by character. "This column is held in a numeric representation" is not. Assert the second. Place it where a change is possible and where being wrong would be expensive: just after the column is built, after any step that rebuilt it from parts, and immediately before the first step whose correctness depends on the column being numeric. Write it in the same file, beside the step, so it fails where it is false rather than in a report. Reading the declared representation is bookkeeping and nearly free; a value-level check is another pass, so spend that one deliberately.
go deeper
Recall that you can state in the file what you expect a column to be, and that the run then stops where that expectation is false rather than much later.
Explain the difference between "these values could be numbers" and "this column is held as numbers", and which of the two a column of digits held as text would pass.
Place the claims deliberately: at each boundary where the representation can change and before the first step that depends on it, weighing what each one costs to evaluate.
Decide what evidence a result must carry before anyone acts on it, and how much of the file's runtime the team will spend proving its own inputs.
## Two claims that are easy to confuse A column invites two quite different assertions, and people write the weaker one by reflex: 1. **"Every value in this column can be read as a number."** A claim about the *contents*. It is answered by looking at the rows. 2. **"This column is held in a numeric representation."** A claim about the *column*. It is answered by reading one recorded property. In this defect the column is digits held as characters, so claim 1 is **true** and claim 2 is **false**. A file guarded only by claim 1 sails straight through, keeps ordering character by character, and produces the wrong total anyway. Claim 2 is the one that fails at the right moment. They are not interchangeable in the other direction either. Claim 2 passing does not mean the numbers are sensible — a numeric column can still be full of impossible values — so a file that cares about both writes both, knowingly, and knows which one each line is buying. ## Where the boundaries are An assertion earns its place where two things are true at once: the representation *can* change there, and something later *depends* on it. That is a short list, and it is much shorter than "every step": - at the point the column first exists in the file, so that everything downstream starts from a stated fact rather than an assumption; - immediately after any step that **rebuilt** the column rather than read it — a fill, a join of a value to a label, a reassembly of the table from parts, an overwrite under the same name; - after any step that can introduce absence into the column, since designs differ in what that does to how the column is held; - immediately before the first step whose correctness depends on the column being numeric — the arithmetic, the threshold comparison, the ordering that will be read as a ranking. The last one matters most and is the one people skip, because by then the column "has obviously been fine for twenty steps". ## What the assertion has to say A useful claim names three things: **which column**, **which representation**, and **at which point**. Vagueness on any of them makes the failure harder to read than the defect: - name the column explicitly, so the failure message is a location and not a puzzle; - state the representation as a requirement, not a preference — the run stops here, it does not warn and continue; - if the column is legitimately allowed to be one of two forms at this point, say so in the claim rather than weakening it to something that can never fail. A claim that cannot fail is a comment with a runtime cost. ## What each claim costs | what you assert | how it is answered | cost on a materialised table | cost on a deferred pipeline | |---|---|---|---| | the declared representation | a recorded property of the column | effectively nothing | usually answerable from the plan, without executing | | every value reads as a number | inspecting every row | a full pass over the column | forces the execution you were deferring | | every value is within an expected range | inspecting every row | a full pass over the column | forces the execution you were deferring | That table is the whole budgeting argument. The representation claim is cheap enough to put at every boundary on the list above without thinking about it. The value-level claims are not, which is why they belong at one or two chosen points rather than sprinkled through the file. ## Where the assertion lives Beside the step, in the same file, on the path that actually runs. Three habits make the difference: 1. **In-file, not in a separate checking pass.** A check that runs afterwards fails at a different time, in a different place, with the intermediate state already gone. 2. **On the real path, not in a copy of the pipeline.** A check that only runs when someone remembers to run it is not evidence. 3. **Failing, not warning.** A warning in a long output is indistinguishable from no warning at all. ## What it cannot do An assertion is a detector, not a guarantee. It says nothing about the steps you did not guard, it does not tell you which step changed the representation — only that something before this point did — and it cannot decide what to do next. Repairing the column means an explicit conversion, and that in turn means choosing up front whether a value that will not convert raises or becomes absent, which is a decision about the data and not a formality. What the assertion does buy is the thing this whole defect class takes away: the failure lands at a point in the file where somebody can still see what went in and what came out.
- Why does a check that every value parses as a number miss this defect?Because the values are digits; they parse perfectly. The column is text made of numerals, so a contents check passes while the column continues to order and compare character by character. The claim that fails is the one about how the column is held, not about what is in it.
- The assertion fails. What are the choices for repairing the column?Narrow it with an explicit conversion, and decide up front what a value that will not convert should do: raise, or become absent. Raising surfaces the awkward values and stops the run; turning them absent keeps the run going and moves the problem into every later aggregate. Neither is a default worth accepting silently.
- Is this worth asserting on a pipeline that defers execution until the end?Yes, and it is usually cheaper there. Designs that build a plan before running it know each step's output representation from the plan, so the claim can be checked without moving any data — and it fails before the expensive execution starts rather than in the middle of it.
saying these in an interview costs you the question
- Checks that the values parse as numbers and calls that a type assertion
- Puts a claim after every step without weighing what each costs
- Asserts only at the end, where the failure says nothing about where
- Keeps the check in a separate file from the step it guards
- Assumes a value-level check costs the same as reading a recorded property