skip to content

In Spark MLlib, what is the difference between a Transformer and an Estimator?

level: juniorimportance: must knowfreq 55%

answer

  1. two abstractions, two method names
  2. one of them must see the data first
  3. fit() versus transform()
  4. fitting yields something that transforms

basics

~20 s

A Transformer implements transform() and converts one DataFrame into another, usually by appending columns. An Estimator implements fit(), which learns from a DataFrame and returns a Model — and that Model is itself a Transformer.

solid answer

~40 s

Spark's DataFrame-based ML API has exactly two component types. A **Transformer** implements `transform(df)` and returns a new DataFrame, normally with extra columns appended — `Tokenizer` appends a words column, `VectorAssembler` appends a `features` vector. An **Estimator** implements `fit(df)` and returns a `Model`, which is itself a Transformer: `LogisticRegression.fit()` returns a `LogisticRegressionModel`, and calling `transform()` on that model appends prediction columns. The split is not "preprocessing versus algorithm". Feature stages that must learn something from the training data are Estimators too — `StringIndexer` learns a label vocabulary, `StandardScaler` learns each column's mean and standard deviation. Stages whose behaviour depends only on their parameters, like `Tokenizer`, `HashingTF` or `VectorAssembler`, are pure Transformers.

code

python · 15 lines
python
from pyspark.ml.feature import Tokenizer, StandardScaler
from pyspark.ml.classification import LogisticRegression

# Transformer: no fit(), transform() appends a column
tokenizer = Tokenizer(inputCol="text", outputCol="words")
words_df = tokenizer.transform(raw_df)

# Estimator: fit() learns, and returns a Model that transforms
scaler = StandardScaler(inputCol="features", outputCol="scaled", withMean=True)
scaler_model = scaler.fit(train_df)      # StandardScalerModel
scaled_df = scaler_model.transform(train_df)

lr = LogisticRegression(maxIter=10, regParam=0.01, featuresCol="scaled")
lr_model = lr.fit(scaled_df)             # LogisticRegressionModel
predictions = lr_model.transform(test_df)

go deeper

for a junior

Recall the two method names and what each returns: transform() hands back a DataFrame, fit() hands back a Model. Being able to add that a Model is itself a Transformer is enough at this level.

for a middle

Be ready to sort real stages into the two camps and justify each. StandardScaler and StringIndexer are Estimators because they learn statistics from the data; Tokenizer and VectorAssembler are pure Transformers driven only by their parameters.

for a senior

Show that the split is the training/serving contract. Anything that is an Estimator holds state learned from the training set, and that fitted state must be persisted and reused at scoring time rather than recomputed on the incoming batch.

for a principal

Frame the contract as the reason feature logic can be versioned as one artifact. Argue for keeping every learned transformation inside the fitted object instead of in ad-hoc SQL that quietly drifts apart between training and serving.

## The two abstractions Spark's DataFrame-based ML API (`org.apache.spark.ml` in Scala, `pyspark.ml` in Python, usually spoken of as "Spark ML" or just MLlib) is built from exactly two kinds of component, both operating on Spark SQL DataFrames. A **Transformer** is anything that turns one DataFrame into another. It implements a single method, `transform(df)`, and it generally works by *appending* columns rather than replacing the frame. `Tokenizer` reads a text column and appends a column of words. `VectorAssembler` reads several numeric, boolean or vector columns and concatenates them into one `features` vector column. A trained model is also a Transformer: `LogisticRegressionModel.transform(df)` reads the `features` column and appends `rawPrediction`, `probability` and `prediction`. An **Estimator** is anything that must look at data before it can do its job. It implements `fit(df)`, and what comes back is not a DataFrame — it is a `Model`, and every `Model` is a Transformer. `LogisticRegression` is an Estimator; `lr.fit(trainingDf)` returns a `LogisticRegressionModel`. The learning algorithm and the learned model are two distinct classes, and the class you are holding tells you exactly what you can do with it. ## Why the split is not "preprocessing versus algorithm" The common mistake is to assume Estimator means "the ML algorithm" and Transformer means "the feature engineering". It does not line up that way. Several feature stages are Estimators, because they must compute something over the training data first: - `StringIndexer` learns the vocabulary of label strings and their frequencies, producing a `StringIndexerModel`. - `StandardScaler` is documented as an Estimator that is fit on a dataset to produce a `StandardScalerModel`; fitting amounts to computing summary statistics (`withStd` is true by default, `withMean` false). - `IDF` learns document frequencies, `MinMaxScaler` learns the per-column minimum and maximum, `CountVectorizer` learns a vocabulary, `Imputer` learns the mean, median or mode it will substitute, `PCA` learns the projection. Others are pure Transformers, because their output for a row is fully determined by their parameters and that row: `Tokenizer`, `RegexTokenizer`, `HashingTF` (hashing needs no learned vocabulary), `Binarizer`, `Bucketizer` (you supply the splits), `SQLTransformer`, `VectorAssembler`. That line is exactly the training/serving contract. Anything on the Estimator side holds state derived from the training set, and that fitted state must be captured once and reused unchanged when scoring — never recomputed from the batch you happen to be scoring. ## Parameters, identity and statelessness Both kinds share one parameter API. A `Param` is a named parameter with self-contained documentation; a `ParamMap` is a set of (parameter, value) pairs. You can set parameters on an instance (`lr.setMaxIter(10)`) or pass a `ParamMap` into `fit()` or `transform()`, in which case the map overrides the setters. Parameters belong to a *specific instance*: every Transformer and Estimator carries a unique id, so a map like `ParamMap(lr1.maxIter -> 10, lr2.maxIter -> 20)` addresses two different logistic regressions without ambiguity. The Spark documentation states that `Transformer.transform()` and `Estimator.fit()` are both stateless. Calling `transform()` twice does not mutate the transformer, and calling `fit()` twice hands you two independent models. All the state that matters lives in the `Model` object that `fit()` returned. ## How the two compose Because a `Model` is a Transformer, and a Transformer is a valid pipeline stage, the two abstractions chain naturally. `Pipeline` is itself an Estimator whose `fit()` produces a `PipelineModel` — a Transformer with the same number of stages, but with every Estimator replaced by the Model it produced. That is the entire design: one contract applied recursively, so a whole workflow becomes a single object you can tune, save and reload. Everything runs on DataFrames, which means Catalyst planning, partitioning and distributed execution serve feature engineering and training alike. It also means the *schema*, not the compiler, validates the wiring — Spark cannot type-check column names at compile time, so pipelines perform runtime schema checking before doing any work. ## In an interview Lead with the method names and the return types: Transformer gives `transform(DataFrame) -> DataFrame`; Estimator gives `fit(DataFrame) -> Model`, and `Model` extends `Transformer`. Then earn the follow-up by naming a feature stage on each side and justifying it — `StandardScaler` is an Estimator because standardising requires the mean, and that mean must come from the training data, not from whatever rows you are scoring today.

  • Is a Spark MLlib Pipeline itself a Transformer or an Estimator?
    A `Pipeline` is an Estimator. Calling `fit()` on it returns a `PipelineModel`, which is a Transformer with the same stages, but with each Estimator stage replaced by the Model that fitting produced. That is why a whole workflow can be handed to `CrossValidator`, saved, and reloaded as one object.
  • Name a Spark MLlib stage that looks like plain preprocessing but is actually an Estimator, and say why.
    `StandardScaler`, `StringIndexer`, `CountVectorizer`, `IDF`, `MinMaxScaler` and `Imputer` are all Estimators, because each must compute something from the training data — a mean and standard deviation, a label vocabulary, document frequencies, a replacement value. `Tokenizer`, `HashingTF`, `Binarizer`, `Bucketizer` and `VectorAssembler` need no such statistic and are pure Transformers.
  • Do fit() and transform() carry state between calls?
    No. Spark documents both as stateless, so calling `transform()` repeatedly does not mutate the transformer and two `fit()` calls give two independent models. The learned state lives in the returned `Model`. Each instance does carry a unique id, which is what lets a `ParamMap` target one specific estimator among several of the same class.

An Estimator is a tailor taking measurements: it has to see you before it can produce anything. The Model it hands back is the finished suit — a Transformer that simply fits, without measuring again.

saying these in an interview costs you the question

  • Says every stage in a Spark MLlib pipeline is a Transformer
  • Thinks fit() returns the transformed DataFrame
  • Calls StandardScaler a Transformer because it only rescales numbers
  • Believes transform() retrains the model on the new data
  • Confuses a fitted PipelineModel with the unfitted Pipeline

context

resolving links…