In t-SNE, what does the perplexity setting actually control?

## The mechanism t-SNE's first step converts the data into probabilities. For each point `i` it computes ``` p(j|i) proportional to exp(-||x_i - x_j||^2 / (2 * sigma_i^2)) ``` over all other points `j`. Note the subscript on `sigma_i`: the Gaussian bandwidth is **per point**, not global. Choosing it is where perplexity enters. For a given `sigma_i`, the distribution `p(.|i)` has a Shannon entropy `H_i = -sum p(j|i) * log2 p(j|i)`. Perplexity is defined as `2^H_i`. For a distribution spread evenly over `k` neighbours and near zero elsewhere, that value is about `k` — which is why perplexity is described as the effective number of neighbours. It is a smooth, weighted count rather than a hard cutoff: a point can have three very close neighbours and a tail of moderately close ones and still hit perplexity 15. The algorithm runs a binary search on `sigma_i` for every point until its perplexity matches the user's setting. Points in dense regions get small bandwidths, points in sparse regions large ones. The setting therefore fixes the *scale of the neighbourhood* the method is asked to preserve, uniformly across the data. ## What changes when you turn it Take the same 5,000-row table. - **Perplexity 5.** Each point is fitted to a handful of neighbours. The map fragments: many small specks, some genuine sub-structure, some pure noise elevated to the status of an island. Fine detail is visible; anything larger than a handful of points is arbitrary. - **Perplexity 50.** Each point is fitted against a broad neighbourhood. Small specks merge, the picture is smoother, fewer and larger groups, and coarse organisation is better retained. Sub-structure inside a group can vanish entirely. Neither picture is the truth. They are two different questions asked of the same data, at two different scales. This is why the standard practice is to sweep several values — say 5, 15, 30 and 50 — and to trust only structure that appears in all of them. A group that exists at one perplexity and nowhere else is a candidate for being an artefact. ## Practical bounds Perplexity must be smaller than the number of points — asking for an effective 50 neighbours out of 30 rows is not meaningful, and implementations will either refuse or produce nonsense. Common guidance is 5 to 50, with larger datasets tolerating and often benefiting from larger values, since a fixed perplexity covers a shrinking fraction of the data as the row count grows. Be clear about what perplexity is **not**: - It is not the number of clusters. It never appears as a cluster count anywhere in the algorithm. - It is not an accuracy knob to be tuned to an optimum. There is no held-out loss to optimise against; the divergence value at convergence is not comparable across settings, because the target distribution `P` itself changes with perplexity. - It does not make the map deterministic. Even at a fixed perplexity, two runs from different random initialisations produce different layouts. ## The UMAP counterpart UMAP does not use perplexity. It builds a weighted graph over each point's `k` nearest neighbours, and that neighbourhood size plays the analogous role: small values emphasise fine local detail, large values smooth the picture and retain more coarse arrangement. It is a hard count of neighbours rather than an entropy-matched effective count, but the tradeoff it exposes is the same one. UMAP's second well-known setting, the minimum distance, is a different animal entirely and is routinely confused with the first. It does not touch the neighbour graph at all: it constrains how closely points are allowed to pack in the final layout. Small values let neighbours collapse into tight, dense clumps, which looks dramatic and is good for spotting fine structure. Larger values spread points out more evenly, which is easier to read and better for seeing overall shape. Changing it changes the aesthetics of the same underlying topology; changing the neighbourhood size changes what the method looked at in the first place. ## How to talk about it in an interview The strong answer names the mechanism (per-point bandwidth tuned by search until the neighbour distribution's entropy matches the target), gives the interpretation (effective number of neighbours), states the tradeoff (local detail against coarse organisation), and finishes with the discipline (sweep several values, believe only what survives). The weak answer calls it a smoothing parameter and stops there.