A topic model's coherence peaks at 18 topics while held-out perplexity keeps improving — how do you choose K?

## Why the two curves disagree This disagreement is the normal case, not an anomaly, and knowing why is the whole answer. **Held-out perplexity** is a predictive-fit measure. You hold out documents (or held-out words within documents), compute the model's average log-probability per held-out word, and report `perplexity = exp(-average log-likelihood per word)`. Lower is better. Adding topics adds parameters and flexibility, so the model can shape the word distribution of unseen text more finely; the curve therefore tends to keep descending, flattening slowly rather than turning up. It is measuring *how well the model predicts word occurrences*, and nothing in that objective rewards a topic being nameable. There is a well-known experimental result behind this: when human raters were asked to spot an intruder word inserted into a topic's top terms, the models that scored best on held-out likelihood were often the ones humans found *least* interpretable. Predictive fit and human-readable themes are genuinely different goals, and optimising the first can hurt the second. **Coherence** was built to close that gap. The common family scores each topic from the co-occurrence statistics of its own top `N` terms: if `credit`, `loan` and `rate` really do turn up in the same documents, the topic is coherent; if a topic's top words never appear together, it is an artefact. One standard form sums, over ordered pairs of top words, `log((co-document count + 1) / document count of the more frequent word)`, then averages across topics. Higher (less negative) is better. Other variants use pointwise mutual information over a sliding window in an external reference corpus. All of them are proxies for readability, and unlike perplexity they do peak: too few topics and each one blends unrelated vocabularies, too many and topics fragment into near-duplicates whose top words are dragged from thin evidence. ## What to actually do 1. **Take the coherence peak as the anchor, not the answer.** A peak at 18 says the neighbourhood of 18 is where topics hang together. Coherence curves are noisy — refit at each K with a couple of seeds before you trust a single point, because a bump of a few percent between K = 16 and K = 18 is often just initialisation. 2. **Read the topics.** This is the step candidates skip and interviewers are listening for. Print the top 10-15 terms per topic for two or three candidate K values and inspect: are there duplicate topics that clearly say the same thing? A junk topic made of boilerplate (signatures, greetings, template text)? A theme you know exists in the corpus that has no topic? Those symptoms tell you to move K, adjust the vocabulary, or change the prior — and no automatic score will tell you which. 3. **Let the downstream task arbitrate if there is one.** If topic proportions feed a classifier, a routing rule, or a retrieval system, that system has a real metric. Sweep K, measure that metric, pick the K that wins. It outranks both proxies because it is not a proxy. 4. **Prefer the smaller K when the curve is flat.** Between two K values with comparable coherence, fewer topics means fewer things for a human to name, maintain, and explain to stakeholders. Interpretability has an ongoing cost, not just a one-off one. 5. **Do not report perplexity as the selection criterion to a business audience.** It is still worth computing as a sanity check — a model whose held-out perplexity is wildly worse than its neighbours is broken — but presenting it as evidence that 60 topics are better than 18 will not survive contact with anyone reading the topics. ## Caveats worth voicing - **Perplexity comparisons must be like-for-like.** Different vocabularies, different held-out splits, or different estimators of the held-out likelihood make the numbers incomparable. If your sweep changed preprocessing along with K, the curve means nothing. - **Coherence has its own failure mode.** It rewards topics built from frequently co-occurring terms, so a topic full of common boilerplate can score beautifully and be useless. That is another reason the eyeball step is not optional. - **The right K may not exist.** If coherence is nearly flat from 10 to 40, the corpus may not have a clean topic structure at all, and forcing a number is a decision about presentation rather than a discovery about the data. Say so rather than manufacturing precision. - **Non-parametric alternatives exist.** A hierarchical Dirichlet process infers the number of topics from the data instead of fixing it, which converts the choice of K into the choice of a concentration parameter. It moves the problem rather than removing it, and adds fitting complexity. ## The one-sentence version Perplexity optimises prediction, coherence approximates readability, and the human reading the topic list is the actual acceptance test — so anchor on coherence, verify by reading, and defer to the downstream metric whenever one exists.