A pressure sensor column contains occasional -999 readings; why is capping them the wrong fix?
answer
- not a measurement at all
- the number is a code
- impossible for the unit
- missing, not extreme
- flag it, then impute
basics
~20 s-999 is a code the logger writes when the sensor is disconnected, not a measurement. Capping folds it into the real data range and teaches a relationship that never existed. Convert it to missing instead.
solid answer
~50 sThe value is not an extreme reading, it is a placeholder, so no outlier rule should touch it. You recognise it by the pattern: hundreds of rows holding one exact repeated value, physically impossible for the unit, with nothing nearby in the distribution. The right move is to convert every -999 to missing, add a boolean indicator saying the reading was absent, and then impute or use a model that consumes missing values directly. Capping it to, say, the 1st percentile would map a disconnected sensor onto a plausible low pressure, so the model quietly learns that outages look like slightly low readings. Dropping the whole row is also wasteful: the other columns in that row are fine, and the fact that the sensor was down is often predictive of the thing you are modelling.
go deeper
Be ready to say that an impossible, often-repeated value like -999 is a placeholder code, and that you turn it into missing before any capping or outlier rule touches the column.
Explain how you detect a sentinel, a spike at one exact value that is impossible for the unit and confirmed in the data dictionary, and why capping quietly converts a status code into a plausible measurement.
Show that you fix the encoding at ingest, keep a missingness indicator because downtime often predicts the target, and audit every numeric column for sentinels before any modelling starts.
Own the standard rather than the incident: a schema and data contract that forbids in-band sentinel codes, plus ingest validation that fails loudly, so no future model has to rediscover the same trap.
## The distinction that matters A numeric column can hold two very different kinds of surprising value. A **genuine extreme** is a real measurement from the tail of the distribution: a pick that really did take 250 minutes, an order that really was large. A **sentinel** is a reserved number that some upstream system writes to mean *no value*, *not applicable* or *device unavailable*, squeezed into the same column because the column has no room for anything but a number. `-999`, `-1`, `9999`, `0` and `1900-01-01` are the classic ones. Outlier treatment applies to the first kind. Sentinels are an **encoding defect**, and they must be fixed before any capping, trimming or robust rule is allowed to run. If you skip that step, every downstream statistic is contaminated: the mean, the standard deviation and every percentile you might have used to set a cap. ## How you spot a sentinel Four signals, in rough order of strength: 1. **One exact value repeats far too often.** Real continuous measurements almost never pile up on a single number. A histogram spike of 300 rows at exactly -999, with an empty gap between it and the rest of the data, is not a tail, it is a code. 2. **The value is impossible for the unit.** Absolute pressure cannot be negative. An age of 999 is not a person. A duration of -1 is not a duration. 3. **The data dictionary or the logging code says so.** This is the cheapest check and the one juniors skip. Ask the team that owns the sensor pipeline. 4. **It correlates with a status.** Sentinel rows cluster inside maintenance windows, or line up with a device-status column that reads `offline`. ## Why capping is specifically the wrong fix Suppose the real readings run 8 to 40 bar and you winsorise at the 1st percentile, which happens to be about 11 bar. Every -999 becomes 11. Three things go wrong. - **You invent data.** Three hundred rows now claim a real measurement of 11 bar that nobody ever measured. - **You teach a false relationship.** If outages happen more often just before a fault, the model now learns that *slightly low pressure* predicts faults, when the true predictor was *no reading at all*. The signal is still there but attached to the wrong cause, and it will break the moment the sentinel code changes. - **You destroy the evidence.** After capping, the -999 rows are indistinguishable from genuine low readings. Nobody downstream can find the bug. Setting the value to the column mean has the same defect with a different number. Leaving it raw is no better: a linear model reads -999 as an enormous negative pressure, a distance-based model puts that row a vast distance from everything, and a tree will happily carve out a branch for *pressure below -100* and learn a real-looking rule from a fiction. ## The correct treatment 1. **Map the sentinel to missing.** Do it as early as possible, ideally at ingest, so no analysis ever sees the raw code. 2. **Add a missingness indicator.** A `pressure_was_missing` flag of 0 or 1 preserves the information that the sensor was down. This is often one of the more useful features in equipment models, because downtime and faults travel together. 3. **Impute the numeric slot** with whatever your missing-value strategy is, or use a learner that accepts missing values natively. 4. **Only then** consider outlier treatment on what remains, because now the percentiles describe real measurements. ## Multiple sentinels and the wider audit Legacy pipelines often use several codes with different meanings: -999 for *sensor offline*, -888 for *not applicable*, 0 for *reading suppressed*. Map all of them to missing, but if the reasons differ and might matter, keep a small categorical reason column so the distinction survives. Never guess which code means what; confirm it. Finally, this is a per-column audit, not a one-off. Before modelling, look at the extreme values of every numeric column, check for value spikes and impossible magnitudes, and confirm the plausible range with someone who knows the source system. Sentinels are the single most common reason that an outlier rule fires on data that has nothing to do with outliers.
- How would you confirm -999 is a sentinel rather than a genuine low reading?Look for a spike of identical values with a gap between them and the rest of the distribution, check whether the number is physically possible for the unit, read the data dictionary or the logging code, and see whether those rows line up with maintenance windows or an offline device-status column. Two of those four agreeing is usually enough.
- Should the fact that the sensor was disconnected be discarded entirely?No. Keep a boolean indicator that the reading was absent. Downtime is frequently informative in equipment and process models, since sensors drop out around interventions and faults. The indicator preserves that signal while imputation fills the numeric slot, and it keeps the two effects separable in the model.
- What if the column uses several sentinels such as -999, -888 and 9999?Map every one of them to missing so no code survives into the numeric column, but if the codes mean different things, record the reason in a small categorical column first. Confirm the meanings against the data dictionary rather than inferring them; guessing that 9999 means the same as -999 can merge two unrelated failure modes.
It is like averaging a thermometer that shows ERR when its battery dies, by deciding ERR means minus a thousand degrees. The display was a status message, not a temperature.
saying these in an interview costs you the question
- Treats -999 as a genuinely low pressure reading
- Caps it to the 1st percentile and moves on
- Deletes the whole row without checking the pattern
- Imputes the mean with no missingness indicator
- Assumes any value far from the mean is an outlier