When does an absolute epsilon comparison of two 64-bit floats fail, and what replaces it?
answer
- a tolerance has to mean something at every scale
- how far apart are neighbouring values at 1e12
- one constant cannot serve 1e-15 and 1e12
- scale the allowance by the operands' magnitude
- keep an absolute floor for values near zero
basics
~20 sA fixed absolute epsilon only works in a narrow magnitude band: near 1e12 adjacent 64-bit floats are further apart than 1e-9, while near zero the same epsilon calls wildly different values equal. Use a relative tolerance with an absolute floor.
solid answer
~50 sThe spacing between adjacent 64-bit floats grows with magnitude: about 2.2e-16 near 1.0, about 1.2e-4 near 1e12, about 0.125 near 1e15. So a 1e-9 absolute tolerance is *stricter than the representation itself* at large magnitudes — two calibration readings one rounding step apart are reported as different, and no amount of correct arithmetic will make them agree. At the other end, 1e-15 and 3e-15 differ by a factor of three yet pass the same test. The production form combines both: `abs(a - b) <= max(absFloor, relEps * max(abs(a), abs(b)))`. The relative term tracks magnitude; the absolute floor keeps the test meaningful near zero, where a pure relative allowance collapses to nothing. Both numbers must be derived — the floor from the instrument's resolution, the relative term from how many rounding steps the pipeline performs — not copied from a snippet.
code
pseudocode · 9 linesEPS = 1e-9
// comparator applied to every calibration pair
diff = abs(a - b)
equal = (diff <= EPS)
...
// a = 1.0e12, b = the next representable value above a
// diff ~= 1.2e-4 -> equal = false (one rounding step apart)
// a = 1.0e-15, b = 3.0e-15
// diff = 2.0e-15 -> equal = true (b is three times a)go deeper
Know that computed floating-point values are compared with a tolerance rather than ==, and that the tolerance is a number someone chose deliberately for that data, not a constant you invent at the call site.
Explain why one fixed epsilon cannot serve all magnitudes, write the combined relative-plus-absolute-floor form from memory, and say which term dominates for large values and which for values near zero.
Demonstrate deriving the floor from the instrument's resolution and the relative term from the pipeline's rounding budget, then centralising the comparator so every call site inherits one reviewed decision.
Own the consequence that tolerance equality is not transitive, so it cannot back sorting, deduplication or keys, and decide where the system quantises onto an exact grid instead of comparing approximate values at all.
## Why one epsilon cannot serve every scale A 64-bit float is `(53-bit integer) x 2^e`, so representable values are spaced by `2^(e-52)` within each binary exponent range. That spacing — the *unit in the last place*, or ULP — grows with magnitude: | magnitude | gap between adjacent 64-bit floats | |---|---| | ~1.0 | 2.2e-16 | | ~1e6 | 1.2e-10 | | ~1e12 | 1.2e-4 | | ~1e15 | 0.125 | Read the table against a 1e-9 tolerance. Near 1.0 it allows about a million rounding steps of slack — very loose. Near 1e6 it is roughly ten steps — reasonable. Near 1e12 it is *less than one step*: two values that differ by a single rounding of the very last bit are already 1.2e-4 apart and fail the test. The comparator has become a bit-equality test wearing a tolerance costume. Near 1e-15 the same epsilon swallows everything, so a residual and a value three times larger are called equal. This is the whole objection to "just compare with epsilon 1e-9". It is not that epsilon comparison is wrong; it is that an *absolute* epsilon encodes an assumption about magnitude that the code never states and the data does not honour. ## The relative form A relative tolerance scales the allowance by the operands: `abs(a - b) <= relEps * max(abs(a), abs(b))`. Now the allowance is a fixed number of rounding steps at any magnitude, which is what "these two computations agree" actually means. It has one hole: near zero the allowance shrinks with the operands. If both values are tiny residuals — the classic output of subtracting two nearly-equal quantities, where the leading digits cancel and only accumulated noise survives — the relative test compares noise against noise and declares them different. And against exactly zero the allowance is exactly zero, so nothing but a bit-identical zero ever matches. ## The combined form, and where the numbers come from ``` abs(a - b) <= max(absFloor, relEps * max(abs(a), abs(b))) ``` The two constants answer two different questions. - **`absFloor` is a domain number.** It is the smallest difference anyone cares about: the instrument's resolution, the noise floor of the sensor, the granularity the calibration report is published at. If the transducer resolves 0.001 units, differences below that are not signal and the comparator should say so. - **`relEps` is a computation number.** Roughly, the number of rounding steps the pipeline performs multiplied by the rounding unit (about 2.2e-16 for this format). A single subtraction earns a tolerance of a few ULP; a long accumulation, a change of units, and a trigonometric conversion earn considerably more. If you cannot state roughly how many operations produced the value, you cannot defend the constant. Write both numbers down with the reasoning beside them. An undocumented epsilon is an outage waiting for the day the data's range changes. ## The ULP formulation The most precise version does not measure the arithmetic difference at all; it counts how many representable values lie between the operands. Because the representable values are laid out so that, for same-signed finite values, their bit patterns interpreted as integers are monotonically ordered, that count is a subtraction on the integer view. This is common in numeric test suites and is exactly "agree to within N rounding steps" with no constant to justify per magnitude. Its awkward cases are comparisons that straddle zero, values in the subnormal range near zero where spacing stops growing, and infinities. The mixed absolute-plus-relative form is easier for a reviewer to reason about, which is why it is the usual production choice. ## Catastrophic cancellation, the reason the floor exists Subtracting two nearly equal values annihilates the leading significant digits and promotes accumulated error into the leading position. Two calibration paths that each compute a large intermediate and subtract can produce residuals whose *relative* error is enormous even though each input was accurate to a rounding step. That is precisely the region where relative tolerance is meaningless and only a domain-derived absolute floor gives a defensible verdict. Where you can, restructure the computation to avoid the subtraction rather than widen the tolerance to hide it. ## The property that surprises people Tolerance-based equality is **not transitive**. `a` can match `b`, `b` can match `c`, and `a` and `c` can be more than a tolerance apart. That is fatal for anything that assumes an equivalence relation: it cannot back a sort's tie-breaking, a deduplication pass, a grouping key, or a hash. When you need a genuine equivalence, quantise onto an explicit grid — round each value to a chosen multiple and compare the exact results — so "equal" becomes a well-defined property of a single value rather than a relation between two.
- Why can a purely relative tolerance not handle a comparison against exactly zero?A relative tolerance scales the allowance by `max(abs(a), abs(b))`. If both operands are zero the allowance collapses to zero, and if both are tiny residuals left over from cancellation the allowance is tiny too, so two values that are both noise are reported as different. Near zero the only meaningful yardstick is an absolute one drawn from the domain: the instrument's resolution, the smallest difference anyone would act on. That is why production comparators keep both terms.
- How do you choose the tolerance numbers instead of copying 1e-9?Derive them from two independent sources. The absolute floor comes from the domain: the smallest difference the instrument resolves or the business cares about. The relative term comes from the computation: approximately the number of rounding steps the pipeline performs times the rounding unit, so a long accumulation earns a looser tolerance than a single subtraction. Record both numbers with the reasoning next to them, because the next person cannot re-derive a bare constant.
- What is a ULP-based comparison and when is it preferable?It counts how many representable values separate the operands rather than measuring their arithmetic difference, so the allowance tracks magnitude automatically with no per-scale constant. It is the tightest formulation for well-behaved same-signed ranges and is common in numeric test suites. It is awkward across zero, in the subnormal range, and with infinities, and it is harder for a reviewer to sanity-check, which is why the absolute-plus-relative form is the usual production default.
- Why can this comparator not be used as the equality behind deduplication?Because it is not transitive. Three readings can be spaced just under a tolerance apart in a chain, so the first matches the second and the second matches the third while the first and third differ by nearly twice the tolerance. Deduplication, grouping and sorting all assume an equivalence relation, and this is not one — the result depends on visit order. When you need a key, quantise each value onto an explicit grid and compare the quantised results exactly.
saying these in an interview costs you the question
- Picks 1e-9 with no reference to the data's magnitude
- Says epsilon comparison is always safe once you add one
- Uses a purely relative tolerance for values that can be zero
- Assumes one tolerance can serve every subsystem
- Believes tolerance-based equality is transitive