When several downstream Spark jobs read one expensive DataFrame, how do you choose between persist() and writing it out?

## Reframe the question: what is the scope of reuse? "Cache or write?" is usually asked as a performance question. It is really a **scope and ownership** question, and the performance answer follows from it. - **Within one Spark application**, cached blocks in executor storage memory are reachable by every subsequent action in that application. Reuse is nearly free. - **Across applications**, they are not reachable at all. Each `spark-submit` gets its own driver and executors; when the application ends, its executors and everything they cached are gone. If your "five downstream jobs" are five scheduled submissions, `persist()` is not a slow option — it is a non-option. So establish first whether the five consumers are five actions in one driver, or five jobs on a scheduler. ## What persist gives you Within one application, persisting an expensive intermediate avoids re-running its lineage for every action. The conditions under which it pays: - The dataset is genuinely read **more than once**. A single linear read-transform-write pipeline gains nothing. - The upstream work is expensive — a shuffle, a join, a heavy UDF, a wide scan. Caching a plain file scan often saves less than it costs. - It **fits** beside execution memory. Cached blocks occupy the storage side of the same unified pool that shuffles and joins draw from; a large cache buys reuse and pays for it in spill. - You control the lifetime. In long-running drivers — notebooks, Thrift servers, streaming applications — forgetting `unpersist()` starves execution memory over hours. And one non-obvious cost: caching creates an optimization barrier. Filters and column pruning cannot be pushed through a cached plan into the source scan, so a `cache()` placed too early in a plan can make the job read substantially more data than the uncached version would. ## What writing gives you Materializing to Parquet (or an open table format) on durable storage costs one serialization and write pass. In exchange: - **Cross-application reuse.** Any job, any schedule, any team can read it. - **Durability.** Executor loss, driver restart and job failure do not destroy it; a retry resumes from the materialized dataset instead of recomputing the pipeline. - **Read efficiency on re-read.** Columnar layout means downstream readers prune columns and push filters, so a consumer that needs three of forty columns reads three. A cached DataFrame is reused whole. - **No memory pressure.** The unified pool stays available for execution. - **Debuggability.** You can query the intermediate, diff it across runs, and reason about it after the fact. The costs are equally real: the write pass itself, storage cost, and — the one teams underestimate — **lifecycle ownership**. A materialized intermediate is a dataset with a path, a schema, a retention policy and a consumer contract. Ten of these accumulated without governance become a warehouse of orphaned staging tables nobody dares delete. ## Where checkpoint() sits `DataFrame.checkpoint()` writes to reliable storage *and* truncates lineage, so the plan no longer references the upstream chain. That matters for iterative algorithms whose plans grow with each iteration until the driver spends its time planning rather than executing, and for very deep lineages where recovery from executor loss would be prohibitive. It is a *mechanism*, not a published dataset: the files are managed by Spark under `spark.sparkContext.setCheckpointDir` and are not something downstream jobs should read. `localCheckpoint()` trades reliability for speed by keeping the truncation on executor local disk — useful for plan-depth control, useless for fault tolerance. ## A decision framework I would actually apply 1. **Consumers in another application or another schedule?** → Write it. Nothing else works. 2. **Consumers in this application, read two or three times, dataset small relative to executor memory, upstream expensive?** → Persist, and unpersist when done. 3. **Consumers in this application but the dataset is large, or memory is already tight, or the stage is already spilling?** → Write it and re-read. Being able to prune columns on the re-read often makes this *faster* than caching, not merely safer. 4. **Plan depth or iteration count is the problem, not reuse?** → Checkpoint. 5. **Read exactly once?** → Do neither. This is the most common unnecessary `cache()` in real codebases. ## The organizational layer At scale, the interesting failure is not a badly chosen storage level; it is a pipeline where the same expensive join is recomputed by six teams because none of them knew the others needed it. That is solved by publishing the intermediate as an owned dataset with a schedule and a contract, not by tuning anyone's cache. Conversely, the opposite failure — every intermediate materialized "just in case" — produces storage cost and a lineage nobody can trace. The judgment being tested is whether you can tell which of those two failure modes your organization is currently in.