In cluster mode on YARN, where do you find the Spark driver's stack trace after a failed job?

## Why the terminal is empty The usual complaint — "the job failed silently" — is a direct consequence of deploy mode. In `--deploy-mode cluster` the driver JVM runs inside the YARN ApplicationMaster container, so its stdout, stderr and log4j output are that container's log files on whichever node YARN chose. The `spark-submit` process is only a launcher; when the application ends it prints the tracking URL, the final status (`SUCCEEDED`/`FAILED`/`KILLED`) and YARN's `Diagnostics:` string, which is usually a truncated one-liner such as `User class threw exception` with no stack trace. Nothing is broken; you are simply reading the wrong process's output. ## While the application is running - **ResourceManager UI → Application → Tracking URL.** For a live application the tracking URL proxies the driver's Spark UI (port 4040 on the driver, reached through the YARN proxy), giving you stages, tasks, SQL plans and the executors tab. - **The AM container log.** The same application page links to "logs" for each attempt's ApplicationMaster container — that is your driver's live stdout/stderr. - **Executors tab.** Each executor row links to its container's stdout/stderr on the owning NodeManager, which is how you reach task-side exceptions without knowing node names. ## After it has finished The Spark UI dies with the driver, so two different systems take over: 1. **YARN log aggregation.** If `yarn.log-aggregation-enable` is `true`, when containers exit the NodeManagers upload their logs into HDFS and you retrieve them with: ```bash yarn logs -applicationId application_1724_0042 -am 1 yarn logs -applicationId application_1724_0042 -containerId container_..._000007 ``` Without aggregation, logs remain in the local `yarn.nodemanager.log-dirs` on each node and are deleted after `yarn.nodemanager.log.retain-seconds`, so you must reach the node before they vanish. This is the single most common reason a post-mortem fails: aggregation was off, or the retention window had passed. 2. **The Spark History Server.** Set `spark.eventLog.enabled=true` and `spark.eventLog.dir` and the driver writes a JSON event log; the History Server (default port 18080) replays it into the familiar Spark UI long after the application is gone. Note what it does *not* contain: it holds Spark's structured events — jobs, stages, tasks, SQL plans, executor add/remove — not your `println`s or a thrown exception's stack trace. For that you still need container logs. ## Reading the failure correctly Work outward from the driver log. A stage that dies from an executor failure will show a driver-side `SparkException: Job aborted due to stage failure` whose message names the *first* failing task and its executor, plus a truncated cause. The real reason often lives in the executor's container log: a `FetchFailedException` on the driver side may be an out-of-memory kill on the executor side, and YARN's own `Container killed by YARN for exceeding memory limits` message appears in the NodeManager log for that container rather than in the Spark log. Get the application ID, pull the AM log first, then pull the named executor's container. ## Practical hygiene - **Print the application ID at submit time and keep it.** Orchestrators should capture it into the task's own log; every later lookup is keyed on it. - **Configure driver logging deliberately.** `spark.driver.extraJavaOptions` can point log4j at a config that keeps a readable level; a driver spamming INFO makes an aggregated log unreadable at 400 MB. - **Turn event logging on everywhere.** It is cheap, and it is the only way to answer "why was this slow" once the job is over. - **On Kubernetes**, the equivalents are `kubectl logs <driver-pod>` (and `kubectl describe pod` for scheduling/pull failures). Note that executor pods are deleted when they terminate unless `spark.kubernetes.executor.deleteOnTermination` is set to `false`, so an executor-side stack trace can disappear within seconds — ship logs off the node if you care about them. - **In client mode none of this applies**: the driver log is your terminal, which is exactly why interactive debugging is easier there and why teams sometimes reproduce a cluster-mode failure in client mode on a smaller input. ## Streaming and long-running jobs For a job that never ends, the History Server shows nothing until it stops. Rely on the live tracking URL, and if the job runs for weeks, consider the incremental event-log settings and a log shipper rather than trying to `yarn logs` a container that is still writing.