Submission and Deploy Modes

## One launcher, every target `spark-submit` (in `$SPARK_HOME/bin`) is the only supported way to start a Spark application, and it is deliberately the same command whether the job runs in a single local JVM, on a Hadoop YARN cluster, on Kubernetes, or on a Spark standalone cluster. It builds the driver's classpath, resolves dependencies, uploads what needs uploading, and asks the chosen cluster manager to start the application. Learning its argument shape is the everyday skill; everything else about Spark deployment is configuration layered on top. The canonical invocation looks like this: ```bash spark-submit \ --class com.example.DailyRollup \ --master yarn \ --deploy-mode cluster \ --executor-memory 8g \ --executor-cores 4 \ --num-executors 20 \ --conf spark.sql.shuffle.partitions=400 \ s3://artifacts/rollup-1.4.0.jar 2026-08-01 ``` ## --master: which cluster manager, and where `--master` is a URL-shaped string naming the resource manager that will hand Spark its containers: - `local`, `local[4]`, `local[*]` — no cluster at all; driver and "executors" are threads in one JVM. This is what unit tests use. - `yarn` — the address comes from the Hadoop configuration (`HADOOP_CONF_DIR`/`YARN_CONF_DIR`), which is why you do not write a host here. - `k8s://https://api-server:6443` — the Kubernetes API server endpoint; Spark creates a driver pod and executor pods. - `spark://host:7077` — a Spark standalone master (7077 is its default port). If you omit `--master`, Spark falls back to `spark.master` from the properties file, and many gateway installations set that for you. Getting `--master` wrong is the classic first-day error: the job silently runs `local[*]` on the edge node and takes forever on data it can barely read. ## --class and the application artifact For a JVM application you package a jar and point `--class` at the fully-qualified class containing `def main(args: Array[String])` (Scala) or `public static void main(String[])` (Java). Spark does not scan the jar for a main class unless the jar's manifest declares one, so the flag is normally required. The jar itself is the final positional argument, and it may be a local path, an HDFS path, or an object-store URI — Spark distributes it to the cluster as part of submission. PySpark inverts this: there is no `--class`. You pass `app.py` as the positional argument, add pure-Python dependencies with `--py-files deps.zip`, and ship a whole packed interpreter with `--archives venv.tar.gz#environment` when the cluster's Python does not match yours. Any JVM connectors a Python job needs (a Kafka or JDBC source, say) still arrive through `--jars` or `--packages` Maven coordinates. Everything after the artifact is application arguments, not Spark arguments. `spark-submit ... app.jar --date 2026-08-01` passes `--date 2026-08-01` to your `main`; putting it before the jar makes `spark-submit` reject it. ## The supporting flags A handful of flags cover most real submissions: - `--deploy-mode client|cluster` — where the driver process itself runs. - `--driver-memory`, `--executor-memory`, `--executor-cores`, `--num-executors` — sizing. - `--jars a.jar,b.jar` — extra jars added to the classpath of **both** driver and executors and distributed for you. - `--packages group:artifact:version` — Maven coordinates resolved at submit time. - `--files config.yaml` — files copied into each executor's working directory. - `--conf spark.some.key=value` — any Spark property; the escape hatch for everything without a dedicated flag. - `--properties-file` — override the default `conf/spark-defaults.conf`. ## Configuration precedence Three layers can set the same property, and they resolve in this order, strongest first: values set programmatically on a `SparkConf`/`SparkSession.builder`, then `spark-submit` flags and `--conf`, then the defaults file. The important exception is anything that configures the driver JVM itself — `spark.driver.memory`, `spark.driver.extraJavaOptions` — in client mode: that JVM is already running by the time your code executes, so the value must come from the command line or the properties file to have any effect. ## What submission actually does `spark-submit` runs a small JVM (`SparkSubmit`) that parses the arguments, resolves `--packages` through Ivy, and then either starts your driver in that same process (client mode) or asks the cluster manager to start it elsewhere (cluster mode). In both cases the application jar, `--jars`, `--files` and `--py-files` are staged so executors can fetch them. Understanding that staging step explains most "class not found on the executor" failures: a path that exists on your laptop means nothing to a container three racks away unless you told `spark-submit` to distribute it.