How does latent Dirichlet allocation model a document as a mixture of topics?

## What LDA is trying to explain Latent Dirichlet allocation is a *generative* probabilistic model of a document collection. Rather than starting from a similarity measure and grouping documents, it writes down a made-up story of how each document could have been produced, then asks: which hidden quantities make the documents I actually observe most probable? Those hidden quantities are the topics. Two objects are latent (unobserved) and both are recovered by fitting: - **Topic-word distributions.** A *topic* is not a name or a label. It is a probability distribution over the whole vocabulary. Topic 3 might put probability 0.04 on `invoice`, 0.03 on `charge`, 0.03 on `refund`, and almost nothing on the other 40,000 words. A human reading the top terms calls it billing; the model itself has no idea what it is called. - **Document-topic distributions.** Each document has a vector of `K` non-negative proportions that sum to 1: how much of this document came from each topic. ## The generative story For a chosen number of topics `K`: 1. For each topic k, draw a word distribution `phi_k` from a Dirichlet prior with concentration `beta`. 2. For each document d, draw a topic mixture `theta_d` from a Dirichlet prior with concentration `alpha`. 3. For each word position in document d: draw a topic index `z` from `theta_d`, then draw the actual word from `phi_z`. That is the whole model. Note step 3: the topic is drawn *per word*, not per document. This is exactly why a document ends up as a mixture — different word positions in the same document can come from different topics, and the document's proportions are just the tally of where its words came from. ## Why a Dirichlet prior A Dirichlet distribution is a distribution over probability vectors that sum to 1 — the natural prior when the thing you are drawing is itself a mixture. Its concentration parameter controls sparsity: - `alpha` well below 1: documents concentrate on a handful of topics (most documents are about one or two things). - `alpha` above 1: documents spread mass evenly across many topics. - `beta` well below 1: topics concentrate on a small set of words, which usually reads as crisper themes. So the priors are not decoration — they are the main non-`K` knobs you have, and they change how interpretable the output looks. ## Inference The posterior over the hidden variables is intractable in closed form, so it is approximated. The two standard families are **collapsed Gibbs sampling**, which repeatedly reassigns each word token to a topic given all the current assignments, and **variational inference**, which fits a simpler tractable distribution to the posterior by optimisation. Both are iterative, both start from a random initialisation, and neither is guaranteed to reach a global optimum. That is worth remembering: two runs on the same corpus give different topics, and their numbering is arbitrary anyway. ## Soft membership is the point The single most useful thing to say in an interview: LDA does not partition documents. A hard clustering forces every ticket into one bucket, which is wrong for a ticket that says *my card was declined so I could not log in and my order never shipped*. LDA returns proportions, so mixed documents stay mixed. Downstream you can still take the argmax topic if you need a single label, but you are throwing information away when you do, and you can see from the proportions whether that argmax was 0.9 (confident) or 0.34 (a coin flip between three themes). ## Assumptions and limits - **Word order is ignored.** LDA treats a document as an exchangeable collection of word occurrences; `bank charged me` and `me charged bank` are the same input. That is why it is cheap and why it cannot model phrases or negation. - **`K` is fixed in advance.** The model will not tell you how many topics exist; it will happily split one real theme into three or merge two into one if you mis-set `K`. Non-parametric extensions such as the hierarchical Dirichlet process infer the count instead, at a cost in complexity. - **Topics are not labels.** Naming a topic from its top words is a human step, and a stakeholder-facing one. - **Small or short documents are hard.** With a few dozen words per document there is little evidence to split among `K` topics, and mixtures come out noisy. ## Scoring a new document Once fitted, the topic-word distributions are frozen and a new unseen document can be scored by inferring only its own topic proportions against those fixed topics. That is what lets a fitted topic model act as a fixed feature extractor: every incoming ticket becomes a `K`-dimensional vector of proportions that a downstream classifier or dashboard can consume.