What is the difference between a generative and a discriminative classifier?

## Two routes to the same decision A classifier is given a feature vector `x` and must choose a label `y`. There are two fundamentally different things you can fit in order to do that, and the split between them is the oldest taxonomy in supervised learning. **Discriminative** methods model `p(y|x)` directly: the probability of each label *given* the features. Some do not even go that far - a support vector machine or a perceptron learns only a decision boundary, a function that says which side of a surface a point falls on, with no probability attached. What all of them share is that training optimises a criterion tied to labelling: get the label right, or get the predicted label probability close to the observed one. Logistic regression, support vector machines, decision trees, random forests, gradient-boosted trees and neural networks are all discriminative. **Generative** methods model the joint distribution `p(x, y)` - how labelled examples as a whole are produced. In practice nobody fits a joint density in one piece; it is factored as ``` p(x, y) = p(y) * p(x|y) ``` so training splits into two easy halves: estimate `p(y)`, how common each class is, and estimate `p(x|y)`, a model of how feature vectors are distributed *inside* each class, fitted from that class's examples only. Prediction then inverts the factorisation: ``` p(y|x) is proportional to p(y) * p(x|y) ``` and you take the label with the largest value. Naive Bayes, linear and quadratic discriminant analysis, and hidden Markov models for sequences are generative. ## Where the name comes from `p(x|y)` is literally a recipe for producing data: draw a class from `p(y)`, then draw a feature vector from that class's distribution, and you have manufactured a new labelled example. That capability - not the ability to write text or paint pictures - is what "generative" means here. Modern generative media models are the same idea scaled up, but in classical ML the word is a statement about *which distribution was estimated*, nothing more. ## What each one spends its effort on The discriminative model spends everything it has on the region where the classes meet. Points deep inside a class contribute little, because moving the boundary does not change how they are labelled. The generative model, by contrast, must describe each class's entire cloud of points - including the parts far from any boundary, which have no influence on the decision at all. That is effort spent on structure the decision does not need, and every assumption it makes about that structure is an assumption that can be wrong. The payoff is that the generative model ends up holding something richer than a decision rule: a description of the data itself. It can produce new feature vectors, evaluate how typical an input is, and re-use `p(x|y)` when the class balance changes. The discriminative model has none of that; it knows only how the label depends on the features. ## Consequences worth having ready - **Probabilities.** Both *can* be probabilistic. Outputting a number between 0 and 1 does not make a model generative - logistic regression does that and is discriminative. - **Labels.** Both need labels to be trained as classifiers. A generative classifier fits one `p(x|y)` per class, which requires knowing which examples belong to which class. - **Sample size.** Generative fits stabilise quickly because their parameters are simple per-class summaries; discriminative fits usually have lower error once enough data arrives. - **New classes.** Adding a class to a generative classifier means fitting one more class-conditional and renormalising the priors; a discriminative model is typically refitted end to end. ## Confusions that cost candidates points *Linear discriminant analysis is discriminative.* It is not. Despite the name, it assumes a normal density per class with a shared covariance matrix, fits those densities plus class priors, and applies the inversion above. The resulting boundary happens to be linear - that is what "linear" in the name refers to - but the object that was fitted is a class-conditional density, which makes the method generative. *Generative means unsupervised.* No. Density estimation without labels is unsupervised; a generative *classifier* is supervised and uses labels to split the data before fitting. *Discriminative means no probabilities.* No. It means the model targets the label given the features rather than the joint distribution. ## Where the line blurs The two routes can arrive at the same *shape* of decision rule. Normal class-conditionals with a shared covariance matrix imply that the log-odds between two classes is a linear function of `x` - exactly the functional form logistic regression fits. The families are still different, because they optimise different objectives from the same data and therefore land on different coefficients. The taxonomy is about what is estimated, not about how curved the boundary is.