The same threshold comparison over a column with absent values yields a two-state result on one tool and a three-state result on another - why?

## What a comparison hands back A comparison written over a whole column is an element-wise operator with a different result type. It walks the operands position by position and allocates a new column the same length, holding a per-position verdict rather than a number. Everything true of arithmetic element-wise operators is true of it: it is one dispatch from your program, the loop runs inside compiled code, and the operands are untouched. The interesting question is what it puts at a position where there is no value to compare. ## Two representations of absence, two answers | | Absence as a borrowed floating-point sentinel | Absence tracked separately | |---|---|---| | How "no value" is stored | a reserved floating-point value in the same buffer | a validity bit, or an equivalent marker, beside a value of the ordinary width | | Result of comparing it to a threshold | false | absent | | Result of comparing it to itself | false - the sentinel is unequal to everything | absent | | States in the condition column | two | three | | What the consumer must handle | nothing extra | what the third state means | The first design's behaviour is not an arbitrary choice: the sentinel it borrows is *defined* by the floating-point specification to compare unequal to every value including itself, so "false at an absent position" falls out of the representation rather than being decided by the library. The second design has no such sentinel to lean on, so it has to decide, and the coherent decision is that a comparison whose input is unknown has an unknown answer. Be careful not to fuse the two axes. **How a tool stores absence** and **what semantics its comparisons run under** are two separate choices, and tools bundle them differently. The pairing above is the common one, not a law. ## Why this is a senior question rather than trivia It is a silent divergence. Both tools accept the expression, both return a column of the right length, neither warns. The difference only shows up in whatever consumes the condition column, and by then the comparison is several steps behind you. Two specific consequences are worth carrying: - **The equality-with-itself trap.** Under a sentinel design, a value at an absent position is not equal to itself. Any code that identifies rows by comparing a column against a copy of itself, or that treats equality as reflexive, breaks exactly at those positions - and reports "different" rather than raising. - **The count that does not add up.** Under a sentinel design, the counts of true and false positions sum to the length of the column. Under three-valued semantics they do not, and the shortfall is exactly the absent positions. That arithmetic is the cheapest way to tell which design you are on without reading any documentation. ## What to check when you move between tools 1. Compare a small column containing an absent value against a threshold, and look at the states in the result. Two states or three answers the question in one line. 2. Compare that column against itself for equality. False at the absent position means a sentinel design; absent means three-valued semantics. 3. Add up the true count and the false count and compare that total against the column's length. A shortfall equal to the absent count is three-valued semantics confirming itself. ## The honest form of the claim Do not say "comparing against absence gives false" and do not say "it gives absence." Say which design you mean: "against a borrowed floating-point sentinel a comparison is simply false, so even equality with itself fails; under three-valued semantics the result is neither true nor false and the condition column carries absence onward." That is the answer that survives the interviewer switching tools on you, and it is also the answer that survives your team switching tools on you. ## What a strong candidate adds unprompted That the operator itself is unremarkable - it is the representation of absence that produces the divergence, not anything special about comparison - and that the result is a new column either way, allocated at the full length of the operands, with the originals unchanged. Locating the surprise in the representation rather than in the operator is the sign that someone understands the mechanism rather than having memorised one tool's behaviour.