A k-means elbow reads k=4 but the silhouette-versus-k sweep peaks at k=7 - how do you choose?

## Why the two routes disagree by design The inertia route scores a partition on **compactness alone**: the sum of squared distances from points to their own centroid. Nothing in it penalises putting two centroids inside one natural group, so the score improves at every k and the only readable signal is the deceleration - the elbow. The silhouette route scores each point on how close it is to its own cluster **relative to** how close it is to the nearest other cluster, then averages. Because over-splitting a real group creates neighbouring clusters that are close by construction, that average genuinely goes down when you split too far, which is why the sweep can have an interior peak at all. So the disagreement is not one method being wrong. It is a compactness-only reading of diminishing returns landing at one granularity, and a separation-aware reading landing at another. The correct mental model is: **the data does not identify k**, and both curves are advisory. ## The checks, in order **1. Are the two sweeps comparable at all?** The single most common cause of a dramatic disagreement is a methodological mismatch: one sweep run on different features, on a differently preprocessed matrix, on a different subsample, or with fewer restarts so its solutions are worse local optima. Re-run both from one protocol before interpreting anything. **2. Is k=7 a refinement of k=4?** Cross-tabulate the two label vectors. If each of the seven clusters falls almost entirely inside a single one of the four, the solutions are nested and there is no contradiction to resolve - you are picking a level on a granularity ladder, and you can honestly present both levels. If instead the seven cut across the four, the two solutions describe genuinely different geometries, which is a sign the structure is weak and the partition is unstable to the method's choices. **3. What are the sizes?** A silhouette average is a mean over points, so it can be lifted substantially by a small number of tightly packed points sitting far from everything else. Look at the seven clusters' sizes. Two clusters of a few dozen points at a large distance are usually outliers or a data-quality artefact (a test account, a bot, a legacy code path). Removing or handling them separately often collapses the k=7 peak entirely - which is the real answer. **4. Does either k survive the intended use?** Ask what each cluster would cause to happen differently. If four of the seven would receive identical handling, the seven-way split is decoration. If two of the four contain sub-populations that genuinely need different handling, the four-way split is under-resolved. ## How to report it Don't present the winner as though the data chose it. Present the sweep, the disagreement, the cross-tabulation and the sizes, then state the choice and its reason: "we ship four because the additional three splits at k=7 subdivide one group and no downstream treatment differs between them". A stakeholder who understands that k was a decision will accept a later revision; one who was told the data said seven will treat any change as an error. ## Common wrong answers - *"Silhouette is more rigorous, so take 7."* Both are heuristics on the same geometry; neither carries a hypothesis test. - *"Average the two and take 5 or 6."* Nothing about the two numbers makes their midpoint meaningful; the resulting partition may score worse on both routes. - *"Run more methods and take a majority vote."* Adding routes that all measure compactness and separation on the same distance metric adds correlated opinions, not independent evidence. - *"Whichever gives prettier clusters."* Selecting on the appearance of the output is how unstable segmentations get shipped. ## When neither is convincing If the elbow is faint and the silhouette peak is both low and driven by small clusters, the honest conclusion is that the separation is weak and any k in the range is defensible. At that point stop asking the curves and choose k from what the clustering is for - with that reasoning written down.