Clustering & Visualization

## What these methods are doing t-SNE and UMAP are *neighbour embedding* methods. Both start by describing, for every point, which other points are its neighbours and how strongly — t-SNE converts distances into probabilities with a bandwidth set by the **perplexity** parameter, UMAP builds a weighted neighbour graph with `n_neighbors` edges per point. Then both search for a 2D (or 3D) layout in which that neighbour structure is reproduced as closely as possible, optimizing by gradient descent from a random or spectral initialization. Nothing in that objective asks the layout to preserve absolute distance, scale, or orientation. The optimizer is free to place a well-separated group anywhere on the canvas as long as its members stay together. That single fact explains almost every misreading of these plots. ## What survives the projection - **Local neighbourhoods.** If two documents are near each other in the plot, they were usually near each other in the embedding space. This is the property the objective actually rewards. - **The existence of well-separated groups.** If the corpus really contains disjoint dense regions, both methods will typically show them as separate blobs. - **A qualitative sense of continuity.** A long smear rather than discrete blobs is a genuine signal that the corpus is a continuum, not a set of clean categories. ## What does not survive - **Distance between clusters.** The empty space between two blobs is not a measure of how different they are. Two blobs at opposite ends of the canvas may be closer in the original space than two adjacent ones. t-SNE is worse at this than UMAP, but UMAP's global geometry is also not metric — treat it as suggestive at best. - **Cluster size and density.** t-SNE in particular equalizes apparent density: a tight cluster of 50 near-duplicates and a diffuse cluster of 5,000 documents can be drawn at similar sizes. Do not read "bigger blob" as "more documents" or "broader topic". - **Axes.** The horizontal and vertical directions have no units and no interpretation. The whole layout can be rotated, flipped or translated with no change in objective value, so "this group is to the left" is not a finding. - **Certainty that a visible group is real.** At small perplexity or small `n_neighbors`, both methods can shatter genuinely uniform data into visually convincing islands. Apparent clusters in noise are a well-known failure of these plots. ## Hyperparameters and stochasticity t-SNE's perplexity roughly sets the effective number of neighbours each point is balanced against; small values emphasize very local structure and fragment the picture, large values smooth it and merge groups. UMAP's `n_neighbors` plays the analogous role, while `min_dist` controls only how tightly points may be packed visually — lowering it makes blobs look crisper without changing what was learned. Both use random initialization and stochastic optimization, so re-running with a new seed changes the layout. The operational consequence: never report structure you have seen in exactly one run at one setting. Re-run across two or three seeds and a couple of neighbour settings. Structure that persists is worth investigating; structure that appears in one panel and vanishes in the next is an artefact of the projection. ## How to use the plot honestly The reliable workflow inverts the naive one. Do the analysis in the original space — cluster there, or compute nearest neighbours there — and use the projection purely as a **display surface**, colouring points by labels or metadata you computed independently. If the independently derived clusters land as coherent regions on the plot, that is mild corroboration; if they are shredded across the canvas, that is a prompt to look harder, not proof either way. A concrete case: a UMAP of a music catalogue's track descriptions. Zooming in, the local neighbourhoods are convincing — acoustic singer-songwriter tracks sit with acoustic singer-songwriter tracks, and pulling up the ten nearest points to any track gives sensible results you can verify by reading them. But the layout also shows two large blobs at opposite corners with a wide void between them, and stakeholders will inevitably ask what the void means. It means nothing. It is the optimizer having found a low-cost arrangement, and a different seed may place those blobs side by side. The verifiable claims from that plot are all local; the eye-catching global ones are not claims at all. ## Alternatives when you need evidence rather than a picture If the question is "are these two groups distinct?", answer it with numbers in the original space: nearest-neighbour purity, the fraction of each group's k nearest neighbours that share its label, or an internal cluster index computed on the full-dimensional vectors. If the question is "what is near this item?", print the ranked neighbour list — it is more informative than the picture and cannot be misread. Save the projection for the slide, and label it as an illustration.