skip to content

Duplicate assays of one water sample differ by 0.3 mg/L - what does that imply for a model predicting the assay?

level: seniorimportance: should knowfreq 45%

answer

  1. same sample, same features, different answer
  2. that gap belongs to the label, not the model
  3. the third term in the decomposition
  4. difference of two replicates is sqrt(2) times one
  5. and check what else varied between runs

basics

~20 s

That disagreement estimates the irreducible-noise term of the error decomposition. No model of the recorded features can beat it on average, so it sets the floor your error should be judged against rather than against zero.

solid answer

~50 s

Two measurements of the same physical sample should agree; the fact that they differ by 0.3 mg/L means the label itself carries variation none of the features explain. That is the noise term in `expected squared error = bias^2 + variance + noise`, and it is a property of the target and the measurement process, not of the model. If differences between duplicate pairs have a standard deviation of about 0.3, a single measurement has a standard deviation near `0.3 / sqrt(2)`, roughly 0.21 mg/L, so an expected squared error below about 0.045 is not achievable and an RMSE near 0.21 already means the model is close to perfect. Practically: report error relative to that floor, stop tuning once you are near it, and consider averaging duplicate assays into the label - averaging two independent measurements halves the noise variance and lowers the floor.

go deeper

for a junior

Recognise that repeated measurements of the same thing disagreeing means the label is noisy, and that no model can predict noise. Naming it as the irreducible term is enough at this level.

for a middle

Convert the disagreement into a number and use it: relate a replicate spread to a variance, and explain why that variance is a lower bound on expected squared error.

for a senior

This is your tier. Show the judgment call - decide when to stop tuning, question whether the replicates really isolate measurement error, and propose redefining the label if a lower floor is worth paying for.

for a principal

Frame the choice as an investment decision: money spent on measurement quality versus money spent on modelling, and what the organisation should promise stakeholders about accuracy given a floor it cannot move.

## What the duplicate disagreement is telling you When the same water sample is split and assayed twice, every feature you would ever put in a model is identical between the two runs: same site, same date, same depth, same everything. Any difference between the two recorded values is therefore, by construction, variation that the features cannot explain. That is the definition of the third term in the squared-error decomposition: ``` expected squared error at x = bias^2 + variance + noise ``` Bias and variance describe your model. Noise describes the target. A model that recovered the true underlying concentration exactly would still miss the recorded label by that much. ## Turning the disagreement into a number If `s` is the standard deviation of a single measurement, and the two duplicates are independent, the difference between them has standard deviation `s * sqrt(2)`. So a typical duplicate gap of 0.3 mg/L implies `s` of roughly `0.3 / 1.41 = 0.21` mg/L, and a noise term of about `0.21^2 = 0.045` in squared-error units. That number is the yardstick. If your model's test MSE is 0.30, roughly 0.045 of it is untouchable and 0.255 is bias plus variance - there is real work left. If the test MSE is 0.055, you have squeezed out almost everything and further tuning is chasing 0.01. Reporting the ratio of achieved error to the floor is far more informative to a stakeholder than an absolute RMSE, because it answers the question they actually have: is there more to get? ## Three moves this unlocks **Stop at the floor.** The most valuable outcome of this calculation is often a decision to stop. Teams burn weeks tuning past the noise floor and then ship a model that looks better on the validation split purely because it fitted that split's noise. **Change the target to lower the floor.** The floor belongs to the *label definition*, and you can redefine the label. If each row's label is the mean of two independent assays instead of a single one, the noise variance halves - the mean of `k` independent measurements has variance `s^2 / k`. Halving the floor by running duplicates is sometimes cheaper than any modelling work. The same logic applies to aggregating a target over a longer window. **Check that the 'noise' is really irreducible.** This is the senior nuance and the one most candidates miss. Irreducible error is only defined relative to the features you have. If duplicate assays are run hours apart and the analyte degrades with time, the disagreement is not pure measurement noise - it is a missing feature (time since sampling, storage temperature) masquerading as noise. Before you accept a floor, ask what varied between the two runs. Genuinely simultaneous replicates give a clean estimate; replicates separated in time or across instruments give an inflated one, and the inflation is exactly the part you could still model away. ## What not to conclude - **Not that the model is done.** The floor is a lower bound on error, not a statement about where your model currently sits. You still have to measure the gap. - **Not that the model is fine because RMSE roughly equals the replicate gap.** Compare like with like: the gap between two measurements is `sqrt(2)` times a single measurement's standard deviation, so comparing RMSE directly to the raw gap flatters the model by about 40%. - **Not that per-row error should be near the floor.** The floor is an expectation. Individual rows will miss by more, and residual plots will still look scattered. - **Not that the floor is the same everywhere.** Measurement error often scales with concentration, so the noise term can be much larger at the high end. If it does, a single global floor will mislead you about which part of the range is worth improving. ## Saying this in an interview The answer that lands has three beats: name the term ("that is the irreducible-noise component of squared error"), quantify it ("a duplicate gap of 0.3 implies a single-measurement standard deviation near 0.21, so an MSE floor near 0.045"), and then show judgment ("before I accept it I would check whether anything besides the instrument varied between the two runs, because that part is a missing feature, not noise"). That last beat is what separates a senior answer from a textbook one.

  • How would you report model quality to a stakeholder once you know the noise floor?
    Report the achieved error alongside the floor, and ideally the fraction of the reducible gap you have closed. Saying 'RMSE 0.24 against a measurement floor of 0.21' answers the question a stakeholder actually has - how much is left - in a way that a bare RMSE never does, and it makes the case for stopping or for investing in better measurement rather than more modelling.
  • Can the irreducible-noise term ever be reduced?
    Not for a fixed target and a fixed feature set - that is what irreducible means. But both are choices. Add a feature that explains part of what looked like noise and it moves into the reducible budget; redefine the label as an average of repeated measurements and the noise variance falls in proportion to how many you average. The term is a floor for a problem definition, not a law of nature.
  • What would make you distrust a noise floor estimated from replicate measurements?
    Anything that differed between the replicates besides the instrument: a gap in time, a different analyst, a different machine, or a sample that changes while it waits. Those inflate the apparent floor with variation that a feature could capture. I would also check whether the spread scales with the measured value, in which case one global floor misrepresents both ends of the range.

saying these in an interview costs you the question

  • Promises to drive error to zero with a better model
  • Compares RMSE to the raw duplicate gap without the sqrt(2) adjustment
  • Treats the noise floor as fixed regardless of available features
  • Blames the whole residual on model bias
  • Never asks what else differed between the two runs

context

resolving links…