What is cophenetic correlation, and how would you use it to compare two linkage methods?

## The cophenetic distance A dendrogram implies a distance of its own. For any two observations `i` and `j`, look up the height of the merge at which they first end up inside the same cluster; that height is the **cophenetic distance** `c(i, j)`. Every pair has one, so a tree over `n` observations induces a full `n x n` matrix of cophenetic distances alongside the original distance matrix `d(i, j)` the tree was built from. That induced distance has a strong structural property: it is ultrametric, meaning that for any three points the two largest of the three cophenetic distances are equal. This is not an accident — it is exactly what being representable as a tree means, and it is also why no dendrogram can reproduce an arbitrary distance matrix. Any tree is a lossy summary of the distances, and the interesting question is how lossy. ## The correlation **Cophenetic correlation** is the Pearson correlation between the `n(n-1)/2` original pairwise distances and the corresponding cophenetic distances. It lies in `[-1, 1]` and in practice sits high — values of 0.7 to 0.9 are routine, so the number is only meaningful in comparison, never in absolute terms. A value of 0.82 is not good or bad; it is good or bad relative to what another linkage rule achieved on the same distances. ## Comparing linkages with it The canonical use is exactly that comparison. Take one distance matrix, build a complete-linkage tree and an average-linkage tree from it, compute the cophenetic correlation of each, and you have a quantitative statement about which tree preserves the original geometry better. This is worth doing because linkage choice is otherwise argued from cluster-shape priors alone, and this gives one measurable axis. Average linkage typically scores highest, and the reason is mechanical rather than mysterious: its merge height is the mean of the cross-cluster distances involved, so heights track the raw distances closely by construction. Complete linkage systematically inflates heights (it reports the farthest pair), single linkage systematically deflates them (it reports the nearest pair), and both distortions cost correlation even when the tree's grouping is perfectly sensible. Ward's heights are not distances between points at all — they are variance increases — so its cophenetic correlation is not on a comparable footing, and comparing Ward against average linkage this way is a category error more often than an insight. ## What it does not tell you This is the part that separates a memorised definition from understanding. Cophenetic correlation scores **the whole tree's faithfulness to the input distances**. It says nothing about: - whether any cut of that tree yields useful groups, - whether the groups it yields are interpretable or actionable, - whether the structure is stable when the data is resampled, - whether the distances the tree was built from were the right distances in the first place. If the inputs encode the wrong notion of similarity, a tree that reproduces them faithfully has faithfully reproduced the wrong thing. The failure mode is concrete: single linkage on chained data can achieve a respectable correlation while producing one sprawling cluster plus a scatter of singletons — a faithful tree that is operationally worthless. Meanwhile a Ward tree with a lower score may be the one that yields the clean, balanced, business-usable groups. Selecting a linkage by cophenetic correlation alone reliably picks average linkage, which is a fine default but did not need a statistic to justify. ## How to use it honestly Treat it as a diagnostic with a narrow claim. Two sensible uses: 1. **Tie-break.** Two linkage rules both give plausible, similarly interpretable trees; take the one that distorts distances less. 2. **Warning flag.** A conspicuously low correlation says the tree is a poor summary of your distances — the data may not be hierarchical in structure at all, and the dendrogram you are about to show a stakeholder implies relationships the data does not support. What it should not do is override interpretability, stability under resampling, or the operational constraints that decide the cut. Presenting a linkage choice as justified because it scored 0.87 against 0.84 is over-reading a comparison that is not that precise.