Clustering and Dimensionality Reduction

A smoothly falling inertia curve with no bend usually means the data is a continuum with no separated groups at any k. Confirm it with the gap statistic, which compares the curve against clusterings of a structureless reference sample and can return k=1.

The Johnson-Lindenstrauss bound sets it from the number of points and the distortion you accept, roughly log(n) divided by epsilon squared. The original 100,000 columns do not appear in the bound at all, and the bound itself is conservative.

Re-cluster many random subsamples of the data and measure how much the resulting partitions agree with each other, usually as the mean pairwise adjusted Rand index over the points any two subsamples share. Tight, high agreement suggests reproducible structure.

Fit a mixture for each candidate component count and score it by a penalised log-likelihood: AIC charges 2 per free parameter, BIC charges log(n). Lower wins, so BIC selects fewer components than AIC.

Silhouette rewards points near their own cluster mates and far from the nearest other cluster. Along a spiral the far end of your own arm is distant while a neighbouring arm is close, so compact round chunks win.

The negative-bar cluster holds points that are on average closer to another cluster, so it overlaps a neighbour. The short-bar cluster sits on a boundary with no space of its own. A respectable-looking average of 0.42 is being carried by the healthy clusters.

k-means minimises total squared distance to cluster centres, so it rewards compact, roughly round groups. An elongated band has large spread along its long axis, so slicing it crosswise lowers that objective.

Mini-batch k-means earns its place when the data will not fit in memory or a full pass per iteration is too slow: it updates centres from small random batches. You pay with a slightly worse, noisier solution.

Agglomerative clustering needs distances between all pairs: 200,000 rows means about 20 billion pairs, roughly 160 GB at 8 bytes each. Practical routes are subsampling and assigning the rest, clustering micro-cluster centroids, or restricting merges to a neighbour graph.

PCA ranks directions by how much the inputs vary, never by how much they predict the label. A discriminative direction with small variance can sit entirely in the discarded 5%, so retained variance and retained signal are different quantities.

Not by itself. Customers moving between segments is the segmentation working, as long as the segment definitions were held fixed. The expensive problem is re-fitting definitions each run, so every segment name silently changes meaning.

The two metrics answer different questions. Perplexity measures held-out predictive fit and usually keeps improving as topics multiply; coherence measures whether a topic's top words really co-occur. If humans read the topics, follow coherence and inspect the top words yourself.

t-SNE has no mapping from feature space to map space: it optimises the coordinates of the rows it was given, so new rows force a full refit and a different layout. UMAP can place new rows into a frozen existing embedding.

Selection cuts what you must collect and keeps every input explainable; extraction keeps signal spread thinly across all 1,000 columns but still requires computing every one of them at serving time. If the latency is upstream, only selection helps.

When the structure is curved. Linear PCA can only rotate axes and drop some, so a manifold that folds back on itself is flattened into an overlapping smear. Kernel PCA works from pairwise similarities, so its components can bend.

Davies-Bouldin averages, over clusters, the worst ratio of two clusters' spreads to the distance between their centroids, and lower is better. Calinski-Harabasz is a between-cluster over within-cluster dispersion ratio, and higher is better. Different aggregations can rank partitions oppositely.

Fuzzy c-means gives every point a membership in every cluster, non-negative and summing to one, so a borderline point reads 0.55/0.45 instead of being forced into one group. The fuzzifier m sets how soft those memberships are.

One global eps encodes a single notion of dense, so it cannot be tight enough for a core and loose enough for a halo. HDBSCAN sweeps all radii instead and keeps the clusters that persist longest.

A component can centre on a few near-identical rows and shrink its covariance toward zero, so their density diverges and the likelihood is unbounded above. Fix it with a covariance floor or a restricted covariance form.

The cophenetic distance between two observations is the height at which they first join in the tree. Cophenetic correlation is the correlation between those tree distances and the original pairwise distances; the higher-scoring tree distorts the input geometry less.

Inspect those 37 records individually first - a group that small is usually outliers, test accounts or a data fault. Either way, no campaign can economically address 37 people, so absorb them into the nearest segment or document them as exclusions.

For t-SNE, no: it produces no reusable mapping, its layout changes run to run, and two coordinates discard almost everything. A frozen UMAP embedding fit only on training folds can be defensible; treat it as a modelling choice.

Judge the top of the ranking, not the whole model. Have experts review the highest-scored items, measure precision at that k, check the ranking is stable across refits, and accept that recall stays unknown until labels accumulate.

The staffing ceiling is a hard constraint and the sweep is advisory, so k is a design decision. Usually the best move is to cluster at the finer k and define a many-to-one mapping from those groups onto the four playbooks, keeping the structure while satisfying operations.

The two numbers answer different questions. Stability says the structure reproduces; a low adjusted Rand index says it does not reproduce that particular taxonomy. Neither is a verdict — the verdict comes from what the clustering was built to support.

Each principal component is a weighted blend of every input ratio, so per-component reasons are useless to an applicant. If the model on top is linear, compose the loadings with its coefficients to recover one exact weight per ratio.

Topic models are not identifiable: random initialisation, a chosen topic count and shifting text mean each refit gives different, differently numbered topics. Freeze one fitted model as the published taxonomy, refit on a deliberate schedule, and review changes with humans.