What does calling fit() on a Spark MLlib Pipeline produce, and how does each stage run?

## What a Pipeline is In Spark's DataFrame-based ML API, a `Pipeline` is an ordered sequence of `PipelineStage`s, where each stage is either a `Transformer` (implements `transform`) or an `Estimator` (implements `fit`). It exists because real ML work is never one call: you tokenize text, vectorise it, maybe scale it, then train. Bundling that sequence into one object is what makes it possible to tune, persist and reload the whole workflow as a unit, and it is what guarantees training data and scoring data pass through *identical* feature processing. Crucially, `Pipeline` is itself an `Estimator`. That is not a curiosity — it is the reason the whole thing can be handed to `CrossValidator`, and the reason `fit()` gives back a fitted object rather than a DataFrame. ## What fit() actually does, stage by stage `pipeline.fit(trainingDf)` runs the stages in the order they were listed, threading a DataFrame through them: - **Transformer stage**: `transform()` is called on the current DataFrame. The stage is carried into the fitted pipeline unchanged, because there is nothing to learn. - **Estimator stage**: `fit()` is called on the current DataFrame, producing a `Model` — a Transformer. *That produced Transformer* becomes the corresponding stage of the resulting `PipelineModel`. If any stages follow, the new Transformer's `transform()` is then applied to the DataFrame before it moves downstream, so later stages see the columns it appends. For the final stage this transform is unnecessary and is not performed. The classic example is `[Tokenizer, HashingTF, LogisticRegression]`. `Tokenizer.transform()` appends a words column. `HashingTF.transform()` turns words into a `features` vector column. `LogisticRegression` is an Estimator, so `fit()` runs on that frame and yields a `LogisticRegressionModel`. ## What comes out: PipelineModel `fit()` returns a `PipelineModel`, which is a Transformer. It has the same number of stages as the original pipeline, but every Estimator has become the Model it produced. At test or scoring time, `pipelineModel.transform(testDf)` sends the data through the fitted stages in order, each one updating the DataFrame and passing it along, and the predictions come out the far end. That symmetry is the whole point. The scaler applies the *training* mean, the indexer applies the *training* vocabulary, and the model applies the coefficients learned alongside them — because they all live in one fitted artifact rather than in code you might edit on one side only. ## How stages are wired Stages are not wired by position. They are wired by column names: a stage's `outputCol` (or `outputCols`) must match the `inputCol` of whatever consumes it. The array order determines execution order, but the data flow is implicit in the column parameters. The examples in the Spark guide are all *linear* pipelines, where each stage consumes the previous stage's output, but non-linear pipelines are legal as long as the data flow forms a directed acyclic graph — and in that case you must list the stages in topological order yourself, because Spark will not reorder them. ## Runtime checking, not compile-time Because pipelines operate on DataFrames with varied types, they cannot use compile-time type checking. Instead, `Pipeline` and `PipelineModel` do *runtime* checking against the DataFrame schema before actually running. This is more useful than it sounds: a typo in an `inputCol`, or an Estimator expecting a vector column that is currently a string, fails up front rather than after forty minutes of shuffling. ## Stages must be distinct instances Each Transformer and Estimator instance carries a unique id, and pipeline stages must be unique instances. Putting the same `myHashingTF` object into a pipeline twice is invalid; two separate instances, `myHashingTF1` and `myHashingTF2`, are fine even though they are the same class, because they get different ids. The ids are also what lets a `ParamMap` name a parameter on one specific stage during hyperparameter search. ## Parameters Stage parameters can be set on the instance (`lr.setMaxIter(10)`) or supplied as a `ParamMap` passed to `fit()`, where the map overrides earlier setters. Because parameters are instance-scoped, a grid can address `hashingTF.numFeatures` and `lr.regParam` in the same search without ambiguity — the mechanism `ParamGridBuilder` and `CrossValidator` are built on. ## What interviewers are checking They want to hear that `fit()` returns a *fitted pipeline*, not just a trained model, and that the feature stages inside it are frozen with their training-time state. Candidates who say "fit trains the model and then I apply my feature code again at scoring time" have described precisely the bug the abstraction exists to prevent.