roadmap
- 01
- 02
- 03
- 04
- 05
- 06
- 07
- 08
Prepare for Apache Spark interview questions on how distributed jobs are planned, shuffled and tuned. Covers DataFrames and RDDs, Spark SQL and Catalyst, partitioning, Structured Streaming and MLlib.
61 of 61 questions in your plan: 0 known, 0 in progress, 0 due, 0 skipped, 61 new. 0 cut, outside your plan.
Spark turns up in more loops than the data engineering one, and each pulls on a different branch of this tree. Pipeline teams ask about batch jobs that already exist and misbehave: a nightly job that now runs past its window, a join that spills to disk, a write that leaves a directory of tiny files. They expect you to reason from what the Spark UI shows back to the plan that produced it. Teams that own streaming ingestion stay on Structured Streaming: what the checkpoint records, when a watermark lets old state go, why some query changes cannot restart from the old checkpoint. Platform and data infrastructure teams care less about the API than about how a job lands on a cluster: where the driver runs in each deploy mode, how executor memory is divided, when caching costs more than it saves. Machine learning engineers and data scientists who build features in PySpark get MLlib pipelines and the price of a Python UDF, which Catalyst cannot look inside and which sends rows out to a Python worker process and back. Nearly every one of those loops opens on the same ground: lazy evaluation, the difference between a transformation and an action, and which operations force a shuffle. A data engineering loop asks about SQL, data modeling and orchestration alongside Spark; for anyone preparing for that whole job, the [Data Engineer roadmap](/roadmaps/data-engineer) sets this topic among those questions.
The [primer on the Spark hub](/topics/data-spark) already explains how these subtopics depend on one another. One pass through it is enough before the drilling starts; reopen it when your answer cites a configuration setting but cannot say what that setting changes in execution. Juniors start with [RDDs and DataFrames](/topics/data-spark-rdds-dataframes): what stays lazy until an action runs, why a DataFrame lets the optimizer do work that an RDD's functions hide from it, and what `collect()` does to the driver. From middle level up, the subtopic that sorts candidates is [jobs, stages and tasks](/topics/data-spark-cluster-tuning-execution-model), because most tuning answers are claims about where a stage boundary falls and how many tasks run inside it; reach it early, with the Spark UI open on a job you ran yourself. What senior interviewers press on hardest is [skew and adaptive execution](/topics/data-spark-partitioning-shuffles-skew-aqe): the one task still running long after its siblings finished, what adaptive query execution can repair at runtime, and when salting the key is still your job. With a week or less, lower the upper end of level until it stops at the seniority the job posting asks for. Above it, interviewers move toward sizing executors and diagnosing memory pressure on a shared cluster, work a hire at your band is rarely handed alone.
Spark turns up in more loops than the data engineering one, and each pulls on a different branch of this tree. Pipeline teams ask about batch jobs that already exist and misbehave: a nightly job that now runs past its window, a join that spills to disk, a write that leaves a directory of tiny files. They expect you to reason from what the Spark UI shows back to the plan that produced it. Teams that own streaming ingestion stay on Structured Streaming: what the checkpoint records, when a watermark lets old state go, why some query changes cannot restart from the old checkpoint. Platform and data infrastructure teams care less about the API than about how a job lands on a cluster: where the driver runs in each deploy mode, how executor memory is divided, when caching costs more than it saves. Machine learning engineers and data scientists who build features in PySpark get MLlib pipelines and the price of a Python UDF, which Catalyst cannot look inside and which sends rows out to a Python worker process and back.
Nearly every one of those loops opens on the same ground: lazy evaluation, the difference between a transformation and an action, and which operations force a shuffle.
A data engineering loop asks about SQL, data modeling and orchestration alongside Spark; for anyone preparing for that whole job, the Data Engineer roadmap sets this topic among those questions.
The primer on the Spark hub already explains how these subtopics depend on one another. One pass through it is enough before the drilling starts; reopen it when your answer cites a configuration setting but cannot say what that setting changes in execution.
Juniors start with RDDs and DataFrames: what stays lazy until an action runs, why a DataFrame lets the optimizer do work that an RDD's functions hide from it, and what collect() does to the driver. From middle level up, the subtopic that sorts candidates is jobs, stages and tasks, because most tuning answers are claims about where a stage boundary falls and how many tasks run inside it; reach it early, with the Spark UI open on a job you ran yourself. What senior interviewers press on hardest is skew and adaptive execution: the one task still running long after its siblings finished, what adaptive query execution can repair at runtime, and when salting the key is still your job.
With a week or less, lower the upper end of level until it stops at the seniority the job posting asks for. Above it, interviewers move toward sizing executors and diagnosing memory pressure on a shared cluster, work a hire at your band is rarely handed alone.