How does k-NN turn the k nearest neighbours into a prediction for classification and for regression?
answer
- retrieve first, then combine
- the k labels are the whole input
- counting for classes, averaging for numbers
- an average cannot leave its own range
- even splits still need a rule
basics
~20 sk-NN finds the k stored examples closest to the query, then aggregates their labels: for classification it predicts the class held by the most neighbours, and for regression it predicts the average of their target values.
solid answer
~50 sGiven a query point, k-NN measures the distance to every stored training example, keeps the `k` smallest, and aggregates only those neighbours' targets. For classification that aggregation is a majority vote: the predicted class is the one appearing most often among the k labels, so with k=5 and neighbours labelled churn, churn, churn, stay, stay the prediction is churn. For regression it is the neighbour mean: the prediction is the arithmetic average of the k neighbour target values, which is why an appraisal built from the eight nearest comparable flat sales in the same postcode is just the mean of those eight prices. Two consequences fall straight out. The prediction is always inside the range of stored targets, so neighbour-mean regression cannot extrapolate. And when the vote splits evenly you need an explicit tie-breaking rule, because argmax over a tied tally is otherwise arbitrary.
go deeper
Be ready to state both rules cleanly in one breath: majority class among the k neighbours for classification, mean of the k neighbour targets for regression. Give a worked mini-example with actual numbers rather than describing it abstractly.
Explain what the aggregation ignores: a plain vote uses distance only to select the k neighbours, never to weight them. Mention the vote share as a coarse probability and the piecewise-constant prediction surface a neighbour mean produces.
Show that you have hit the operational edges: an undocumented tie-breaking rule makes predictions irreproducible, and a neighbour mean silently caps predictions at the observed target range, which shows up as systematic under-prediction at the top end.
Own the framing question: neighbour aggregation gives you no coefficients and no extrapolation, so choosing it commits the team to a model nobody can explain by inspection and that quietly degrades whenever the deployed input range drifts past the stored data.
## The rule in one line k-Nearest-Neighbour prediction has two steps. **Retrieve**: measure the distance from the query point to each stored training example under some distance function and keep the `k` closest ones. **Aggregate**: combine the targets of exactly those `k` neighbours into one prediction. Everything else about the method is a variation on the second step. ## Classification: the majority vote Let the k retrieved neighbours carry labels `y_1 ... y_k`. The plain vote counts how many neighbours belong to each class and returns the largest count: ``` count(c) = number of neighbours whose label is c prediction = argmax over c of count(c) ``` With `k = 7` and neighbours `{approve, approve, approve, approve, decline, decline, decline}` the tally is approve 4, decline 3, so the prediction is `approve`. Nothing about how *close* those four approvals were enters the calculation — under a plain vote every one of the k neighbours has exactly one vote, whether it sits almost on top of the query or at the far edge of the neighbourhood. A useful by-product is the **vote share**: `count(c) / k` is a crude estimate of the probability that the query belongs to class `c`. It is coarse, because with `k = 5` the only values it can ever take are 0, 0.2, 0.4, 0.6, 0.8 and 1.0. ## Regression: the neighbour mean For a continuous target the aggregation is an average instead of a count: ``` prediction = (1/k) * sum of y_i over the k neighbours ``` Appraising a flat from the eight nearest comparable sales in the same postcode is exactly this: no price model is fitted, no coefficients exist; the answer is the mean of eight stored sale prices. Some variants use the **median** of the k targets instead, which is more robust when one neighbour carries an outlying target, at the cost of ignoring the size of the other values. Two properties of the mean matter in interviews: - **No extrapolation.** An average of stored targets always lies between the smallest and largest of them, so the model can never predict a price above the highest comparable sale or a demand above the highest observed demand. A linear model can; a neighbour mean cannot. - **Flat prediction surface between points.** The prediction changes only when the *set* of k neighbours changes, so the fitted surface is piecewise constant, with jumps where the neighbour set swaps over. ## Ties, and why they are not an afterthought The vote can end level. With a binary problem and an even `k`, a 3/3 split is possible; with three or more classes an odd `k` does not save you either — six neighbours across three ticket-priority classes can come back 2/2/2. Common tie-breaking rules, roughly in order of how principled they are: 1. **Fall back to the nearest** tied neighbour's class — decide by proximity rather than by count. 2. **Break with distance-weighted totals**, which are almost never exactly equal. 3. **Reduce k by one** and re-vote, which resolves binary ties but can still tie in multi-class problems. 4. **Prefer the class with the higher training prevalence**, or a fixed order, which is deterministic but arbitrary. Whatever the rule, it must be *fixed and documented*. A tie broken by whichever label the sort happened to put first is a silent source of irreproducible predictions. ## What the aggregation does not do The aggregation step does not care how the neighbours were found, and it does not repair anything upstream of it. If one feature is measured in metres and another in millimetres, the retrieved neighbours will be the wrong points, and a perfectly executed majority vote over the wrong points is still wrong. Similarly, the vote takes distances only as an ordering: a plain vote among the k nearest treats a neighbour at distance 0.1 and one at distance 10 identically, which is precisely the weakness that distance-weighted voting is designed to remove. ## Sanity checks worth remembering - `k = 1` reduces both rules to "copy the nearest stored target": the vote has one voter, and the mean of one value is that value. - If `k` equals the number of training points, the vote returns the overall majority class and the neighbour mean returns the global average target, for **every** query — the input stops mattering. - The prediction is defined only relative to the stored data; there are no learned coefficients to inspect, so "what does the model say the effect of income is?" has no answer in this method.
- With k=6 and three support-ticket priority classes the vote comes back 2/2/2 — how do you break the tie?Pick a deterministic rule and document it. The most defensible is to decide by proximity: take the class of the closest neighbour among the tied classes, or equivalently re-run the tally with distance weights, which almost never ties exactly. Reducing k by one is a common alternative but can tie again in a multi-class problem. What matters is that the rule is fixed, not that a sort order silently decides it.
- Why can neighbour-mean regression never predict a value above the largest target in the training set?Because the prediction is an arithmetic mean of k stored target values, and a mean always lies between the minimum and maximum of the values averaged. The method has no slope or intercept to project beyond observed data, so it cannot extrapolate. If the deployed range genuinely extends past the training range — forecasting demand above anything ever seen — a neighbour mean will systematically under-predict there.
- What does the fraction of neighbours voting for a class give you beyond the hard label?A crude probability estimate: with k=10 and 3 neighbours in the positive class, the vote share is 0.3. It is useful for ranking cases rather than just labelling them, but it is coarse — with small k it can only take k+1 distinct values — and it is unweighted, so a neighbour on the far edge of the neighbourhood contributes as much as one sitting on the query.
It is how a surveyor prices a flat: find the eight most comparable recent sales nearby and average them, rather than fitting a formula for what a square metre is worth.
saying these in an interview costs you the question
- Averages the k distances instead of the k neighbour targets
- Thinks k is the number of classes or the number of features
- Claims regression takes a majority vote over target values
- Believes an odd k removes ties in a multi-class problem
- Expects neighbour-mean regression to extrapolate beyond observed targets