Every retrain of a production topic model returns different topics — how do you deliver something stakeholders can trust?

## Diagnose before you fix Three distinct sources of movement get conflated, and separating them is most of the answer. 1. **Non-identifiability of the fit.** Inference for topic models starts from a random initialisation and converges to a local optimum among many of near-equal quality. Topic numbering is arbitrary — there is no reason run 2's topic 7 corresponds to run 1's topic 7 — and beyond relabelling, the actual partition of vocabulary can genuinely differ. Fixing the random seed makes a run reproducible, which is necessary hygiene, but it does not make the result *stable*; it just picks the same arbitrary answer every time, and it stops being the same answer the moment the data or the topic count changes. 2. **Corpus drift.** Real document streams change: new products, new incident types, seasonal vocabulary. Some of the topic movement is the model correctly tracking a world that moved. This is the movement you want to see, and it is why the answer is not simply 'never refit'. 3. **Sensitivity to settings.** The number of topics and the prior concentrations shift where boundaries fall. Two topics that were separate at one setting merge at another, so a change of configuration between runs will look to a stakeholder exactly like model instability. ## The operating pattern that works **Separate the taxonomy from the model.** The deliverable stakeholders care about is a stable set of named themes. Make that an artefact you own, versioned in one place, with human-written names and definitions. The topic model produced version 1; it does not get to silently rewrite it every night. **Freeze and score.** Fit once, have humans review and name the topics, then freeze the topic-word distributions. New documents are scored against those frozen topics, which is cheap and gives identical results for identical inputs forever. Day-to-day reporting is then perfectly stable, and the only moving part is the schedule on which you decide to revisit. **Make refitting a reviewed event.** On a cadence — quarterly is a reasonable default for most content streams — fit a candidate model on refreshed data and produce a *diff*, not a replacement. Align candidate topics to incumbent topics by similarity between their word distributions, then report: which topics matched cleanly, which incumbent topic split into two, which two merged, which candidate topic has no counterpart (a genuinely new theme), and which incumbent topic has no counterpart (a theme that died). Humans approve that diff, keep the old names where topics matched, and name only the new ones. That gives stakeholders continuity through change instead of a fresh set of unfamiliar labels. **Monitor for the thing a frozen model cannot see.** A frozen model forces every new document into old topics; a genuinely new theme is silently smeared across them rather than announced. So track the signal that reveals it: the share of documents whose top topic proportion is unusually low, and the rate of out-of-vocabulary or newly frequent terms. A rise in either is the trigger to refit early rather than wait for the calendar. **Consider that topics were the wrong deliverable.** This is the judgment an interviewer is really probing. Unsupervised topic models are a *discovery* instrument: excellent for the question 'what is in this pile of documents that we did not know about?'. They are a poor *reporting* instrument, because reporting demands a fixed set of categories with agreed definitions and consistent counts quarter over quarter. Once the organisation has settled on a taxonomy, the honest tool is a supervised classifier trained against that taxonomy: labels are stable by definition, accuracy is measurable per class, and adding a category is an explicit decision with a cost. The mature arc is topic model to write the taxonomy, human curation to fix it, supervised model to apply it, topic model again on residual or low-confidence documents to find what the taxonomy is missing. ## What to say about stability numbers If asked to *quantify* instability, describe fitting several models with different seeds and measuring how consistently the same topics reappear — for example, the fraction of a topic's top terms that recur in its best-matching topic from another run, averaged over topics. A model whose topics reappear across seeds is one you can defend; one whose topics scatter is telling you the corpus does not support that many themes, or that the documents are too short for the evidence required. Reporting that fragility up front, before a stakeholder discovers it by noticing last quarter's chart no longer reconciles, is the difference between a model people trust and one they quietly stop using. ## Anti-patterns to name - Refitting nightly and regenerating topic names from each run's top words, so the dashboard's categories silently change meaning. - Fixing the seed and declaring the problem solved, without ever asking whether the topics reproduce under a different seed. - Raising the topic count until labels 'settle', which fragments themes rather than stabilising them. - Averaging topic word distributions across runs without aligning them first, which blends unrelated topics into mush.