How do you save and reload a fitted Spark MLlib PipelineModel, and what are the limits?

## The API ML persistence was added to the Pipeline API in Spark 1.6, and as of Spark 2.3 the DataFrame-based API in `spark.ml` and `pyspark.ml` has complete coverage — every standard estimator, model and pipeline can be written and read. The write side is `model.write().overwrite().save(path)`; without `overwrite()` an existing path is an error. Both fitted and unfitted objects are savable: a `Pipeline` (the recipe) and a `PipelineModel` (the recipe plus everything learned). The read side must use the class that matches what was written — `PipelineModel.load(path)` for a fitted pipeline, `Pipeline.load(path)` for an unfitted one. Loading a fitted pipeline with `Pipeline.load` is a common and confusing error. ML persistence works across Scala, Java and Python: a pipeline trained in PySpark loads in a Scala job, which is exactly what teams want when data scientists train in Python and the production scoring job is JVM code. R is the exception — it currently uses a modified format, so models saved in R can only be loaded back in R (tracked as SPARK-15572). ## What is on disk `save()` writes a **directory**, not a single serialized blob, and it writes through Spark's filesystem layer, so the path can be local, HDFS, or object storage. Inside are pipeline-level metadata (the uid, the stage ordering, the Spark version that wrote it) and a `stages` subdirectory containing one entry per stage. Each stage entry holds its own metadata — the stage class and its parameter values — and, for fitted stages, a data portion carrying the learned content: coefficients, a label vocabulary, a scaler's means and standard deviations, tree structures. That data is written in Spark's own formats, which is why the whole thing must be read back through Spark rather than parsed by hand. Two consequences follow. First, you cannot treat a saved model as a file to attach somewhere; copying it means copying a directory tree. Second, everything the pipeline learned is in there, which is precisely the point — the artifact is self-contained and cannot drift away from the feature code that produced it. ## Compatibility, stated precisely Spark's documented guarantees are worth quoting accurately, because candidates routinely overstate them. *Can a model saved by Spark X be loaded by Spark Y?* Across **major** versions: no guarantees, but best-effort. Across **minor and patch** versions: yes, these are backwards compatible. And there are no guarantees of a stable persistence *format* — the guarantee is about model loading being backwards compatible, not about the bytes staying the same. *Will it behave identically?* Across major versions, again no guarantees but best-effort. Across minor and patch versions, identical behaviour except for bug fixes. Any breaking change across a minor or patch version is reported in the release notes; if a breakage is not reported there, it should be treated as a bug. Operationally this means a Spark major upgrade needs a plan for models in flight. Load every production model against the new version in a staging job before the upgrade, keep the training code and the training data snapshot so a model can be re-fitted rather than rescued, and record the Spark version alongside every stored artifact. ## Custom stages If your pipeline contains a stage you wrote, that stage must be persistable or the whole pipeline fails to save — one unwritable stage poisons the entire artifact. In practice a custom transformer needs to implement Spark's writable/readable traits (in PySpark, `DefaultParamsWritable` and `DefaultParamsReadable`, with all its state expressed as `Param`s rather than ordinary attributes). A Python-only custom stage will also not load in a Scala job, which quietly breaks the cross-language promise for that pipeline. This is a strong argument for expressing custom logic with `SQLTransformer` or built-in stages wherever the logic allows it. ## The serving limit A loaded `PipelineModel` scores by `transform()` on a DataFrame. That is a distributed batch operation: it needs an active `SparkSession`, it plans a job, and it launches tasks. For scoring millions of rows nightly that is ideal. For answering one HTTP request in a few milliseconds it is the wrong tool by an order of magnitude — the job-launch overhead alone dwarfs the request budget. The realistic options are to batch-score into a serving store (a key-value store or an online feature store) and serve lookups, or to export the model into a non-Spark runtime. Spark's own export support is thin: `toPMML` exists only on the RDD-based `spark.mllib` side and covers a short list of models — `KMeansModel`, `LinearRegressionModel`, `RidgeRegressionModel`, `LassoModel`, `SVMModel` and binary `LogisticRegressionModel`. For unsupported models there is either no `toPMML` method or an `IllegalArgumentException`. Anything richer means a third-party exporter, and that is an architectural decision to take before training, not after. ## How to answer Give the two calls and the fact that the output is a directory. State the compatibility rule as minor/patch guaranteed, major best-effort. Then show operational judgment: name the custom-stage trap, and say plainly that `transform()` is a batch path so online serving needs batch scoring or an export.