Two million rows carry a quantity column where a few dozen entries are not quantities; which disposition do you take, and what does each cost downstream?

seniorshould knowfreq 46%

answer

look at the distinct offending values first
damage argues one way, domain the other
loud and early beats quiet and late
decide at the boundary, report a count

basics

~20 s

Decide at the boundary, not downstream. Refusing stops the batch at a known line and keeps the column typed; falling back keeps the batch running and taxes every later pass; a tagged union keeps the evidence and makes every consumer branch.

solid answer

~50 s

First find out what the few dozen entries are, because the disposition follows from that. A recurring placeholder word, a unit suffix or a sentinel from an older system is domain information and argues for a declared union — the kinds are bounded and meaningful. Unrepeatable garbage argues for refusal, which stops the run at a known place and keeps the cost next to whoever can fix the source. The one thing not to do is let the column fall back to holding anything by default, because that is the option nobody chose: the batch completes, the report looks plausible, and every pass over that column pays per-value dispatch until someone investigates a slowdown weeks later. Also fix where the decision lives: one place at the boundary, applied on every batch, not a repair improvised in whichever notebook noticed first.

First find out what the few dozen entries **are**, because the disposition follows from that. A recurring placeholder word, a unit suffix or a sentinel from an older system is domain information and argues for a declared union — the kinds are bounded and meaningful. Unrepeatable garbage argues for refusal, which stops the run at a known place and keeps the cost next to whoever can fix the source. The one thing not to do is let the column fall back to holding anything by default, because that is the option nobody chose: the batch completes, the report looks plausible, and every pass over that column pays per-value dispatch until someone investigates a slowdown weeks later. Also fix **where** the decision lives: one place at the boundary, applied on every batch, not a repair improvised in whichever notebook noticed first.

go deeper

for a junior

Recall that there is a choice here at all, and that letting it happen by itself is also a choice. Ask what the odd values actually say before doing anything.

for a middle

Be able to price all three outcomes: what the run does, what later steps inherit, and why an untyped column costs on every pass and not just once.

for a senior

Demonstrate the diagnosis: distinct offending values, concentration, whether it is new, and who can fix the producer — then place one decision at the boundary and require it to report a count.

for a principal

The standing call is which disposition is the default for every column the platform builds, and what a loud nightly failure is worth against a silently plausible number. Those costs land on different teams, which is why the decision is owned rather than improvised.

Start with what the misfits are, not with what to do

A few dozen non-conforming entries in two million rows is not one situation, it is three, and they call for different dispositions. Before choosing, answer four questions:

What are they? Look at the distinct non-conforming values, not a sample of rows. A few dozen entries usually collapse to two or three distinct strings, and those strings are the answer: a placeholder word, a unit or currency suffix, a sentinel from an older system, or genuine garbage.
Are they concentrated? All from one source, one day or one key range is a different diagnosis from a steady trickle across the whole set.
Is this new? A first appearance argues that something changed upstream. A steady presence every batch means it is part of the data, whatever anyone intended.
Who can change the producer? If nobody can, refusal turns into a standing outage and a different answer wins.

The three dispositions, priced

disposition	what the run does	what downstream inherits	when it is right
Refuse	stops at the value that did not fit, with the value in hand	nothing — the column never existed in a bad state	the misfits are damage, and someone can fix the source
Fall back to holding anything	completes, silently	an untyped column: per-value dispatch on every pass, comparisons without a single rule, aggregates over a subset the engine chose	almost never on purpose; it is what you get by default
Declared union	completes, with each row tagged as to its kind	a branch at every read site, and no surprises	the extra kinds are expected, bounded and meaningful

The asymmetry worth stating out loud: refusal fails loudly and early, the fallback fails quietly and late, and those are not two points on one scale. A stopped batch costs hours of one team's attention. A plausible wrong number costs whatever was decided on the strength of it, discovered by accident.

Where the decision belongs

Wherever the table is built, once, for every batch — not in the step that happened to notice. Three reasons:

Different consumers otherwise make different choices about the same rows, and two reports disagree for reasons nobody can reconstruct.
A repair applied where the problem was noticed runs after the damage has spread into intermediate results.
A single decision point is the only place a declared column type can be enforced at all, because it is the only place every value passes through.

And whichever disposition you take, it must be recorded in the run's output: how many values did not conform, what the distinct offending values were, and which batch. A disposition without a count is indistinguishable from the same disposition silently firing on ten thousand rows next month.

Reversibility

The three differ in what they leave you able to change your mind about:

Refusal keeps the original input intact and unprocessed. It is entirely reversible — you can re-run with a different disposition once you know more. It is the strongest position to be in while still deciding.
A declared union keeps both the value and its kind, so a later pass can re-decide per kind without going back to the source.
The fallback keeps the values but loses every guarantee about them, and the column is now a thing you inspect rather than a thing you know. Worse, anything computed from it has already been published.

That ordering is the argument for treating refusal as the default position for an unexplained mixture and relaxing it deliberately, rather than the other way round.

What the interviewer is testing

They want to see that you do not answer immediately. The weak answer is a technique — a conversion, a cleanup step — applied before anyone has looked at what the offending values say. The strong answer inspects the distinct values first, distinguishes damage from domain, names all three dispositions with their downstream price, puts the decision at the boundary rather than in the consumer, and insists that whichever one is chosen reports a count. The candidate who has lived through this also says which outcome the current system produces by default, because the default is what happens every night when nobody is choosing anything.

## Start with what the misfits are, not with what to do A few dozen non-conforming entries in two million rows is not one situation, it is three, and they call for different dispositions. Before choosing, answer four questions: 1. **What are they?** Look at the distinct non-conforming values, not a sample of rows. A few dozen entries usually collapse to two or three distinct strings, and those strings are the answer: a placeholder word, a unit or currency suffix, a sentinel from an older system, or genuine garbage. 2. **Are they concentrated?** All from one source, one day or one key range is a different diagnosis from a steady trickle across the whole set. 3. **Is this new?** A first appearance argues that something changed upstream. A steady presence every batch means it is part of the data, whatever anyone intended. 4. **Who can change the producer?** If nobody can, refusal turns into a standing outage and a different answer wins. ## The three dispositions, priced | disposition | what the run does | what downstream inherits | when it is right | |---|---|---|---| | **Refuse** | stops at the value that did not fit, with the value in hand | nothing — the column never existed in a bad state | the misfits are damage, and someone can fix the source | | **Fall back to holding anything** | completes, silently | an untyped column: per-value dispatch on every pass, comparisons without a single rule, aggregates over a subset the engine chose | almost never on purpose; it is what you get by default | | **Declared union** | completes, with each row tagged as to its kind | a branch at every read site, and no surprises | the extra kinds are expected, bounded and meaningful | The asymmetry worth stating out loud: refusal fails **loudly and early**, the fallback fails **quietly and late**, and those are not two points on one scale. A stopped batch costs hours of one team's attention. A plausible wrong number costs whatever was decided on the strength of it, discovered by accident. ## Where the decision belongs Wherever the table is built, once, for every batch — not in the step that happened to notice. Three reasons: - Different consumers otherwise make different choices about the same rows, and two reports disagree for reasons nobody can reconstruct. - A repair applied where the problem was noticed runs after the damage has spread into intermediate results. - A single decision point is the only place a declared column type can be enforced at all, because it is the only place every value passes through. And whichever disposition you take, it must be **recorded in the run's output**: how many values did not conform, what the distinct offending values were, and which batch. A disposition without a count is indistinguishable from the same disposition silently firing on ten thousand rows next month. ## Reversibility The three differ in what they leave you able to change your mind about: - **Refusal** keeps the original input intact and unprocessed. It is entirely reversible — you can re-run with a different disposition once you know more. It is the strongest position to be in while still deciding. - **A declared union** keeps both the value and its kind, so a later pass can re-decide per kind without going back to the source. - **The fallback** keeps the values but loses every guarantee about them, and the column is now a thing you inspect rather than a thing you know. Worse, anything computed from it has already been published. That ordering is the argument for treating refusal as the default position for an unexplained mixture and relaxing it deliberately, rather than the other way round. ## What the interviewer is testing They want to see that you do not answer immediately. The weak answer is a technique — a conversion, a cleanup step — applied before anyone has looked at what the offending values say. The strong answer inspects the distinct values first, distinguishes damage from domain, names all three dispositions with their downstream price, puts the decision at the boundary rather than in the consumer, and insists that whichever one is chosen reports a count. The candidate who has lived through this also says which outcome the current system produces **by default**, because the default is what happens every night when nobody is choosing anything.

Why is the default outcome the most dangerous of the three?
Because nobody chose it and nothing reports it. The batch completes, the numbers look plausible, and the column quietly costs per-value dispatch on every pass afterwards. The fault is discovered as a performance mystery or a wrong figure weeks later, long after the batch that introduced it has been forgotten.
The producing team cannot be changed and refusal would mean nightly outages. Now what?
Then the mixture is part of your reality, so make it explicit rather than implicit: bound the kinds you will accept, tag each row, and refuse anything outside that set. You get a running pipeline that still fails on genuinely new kinds, instead of one that absorbs everything silently.
What must the run report regardless of which disposition you choose?
A count of non-conforming values, the distinct offending values, and which batch they came from. Without a count, the same disposition firing on thirty rows and on thirty thousand looks identical, and the day the volume changes is the day nobody notices.
Why inspect the distinct offending values rather than a sample of rows?
Because a few dozen bad entries usually reduce to two or three distinct strings, and those strings carry the diagnosis: a placeholder, a unit suffix, a legacy sentinel, or noise. A row sample shows you volume; the distinct set shows you cause, and the cause chooses the disposition.

saying these in an interview costs you the question

Picks a repair before looking at the distinct offending values
Treats the silent fallback as an acceptable default
Applies the decision in the consumer rather than at the boundary
Never asks whether the entries are damage or domain
Takes a disposition without reporting how many values it hit
Assumes refusal is always safe even where nobody can fix the producer

Two million rows carry a quantity column where a few dozen entries are not quantities; which disposition do you take, and what does each cost downstream?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Start with what the misfits are, not with what to do

The three dispositions, priced

Where the decision belongs

Reversibility

What the interviewer is testing

▾common follow-ups4 questions

▾red flags6 flags

Start with what the misfits are, not with what to do

The three dispositions, priced

Where the decision belongs

Reversibility

What the interviewer is testing

answer

go deeper