In an AWS Glue Spark job, what does moving from G.1X to G.2X workers change?

## The sizing model A Glue Spark job's capacity is two numbers: which **worker type** and how many **workers**. Everything else — executors, cores per executor, memory — follows from that; you do not hand-tune `--num-executors` the way you would with `spark-submit`. The unit underneath is the **DPU**, a Data Processing Unit, which bundles vCPU and memory together: - **G.1X** — 1 DPU per worker. - **G.2X** — 2 DPUs per worker: double the vCPUs, double the memory, more attached disk. - **G.4X / G.8X** — larger still, for jobs whose per-task working set genuinely needs it. - **G.025X** — a fractional worker intended for low-volume streaming jobs. Billing is by DPU-hours consumed, so `worker_type_dpus × number_of_workers × duration` is the shape of the cost. That is why the substitution matters: 10 × G.2X and 20 × G.1X are the same capacity purchase with very different runtime characteristics. Note also that the capacity you request includes the Spark driver, not only executors. ## What going up a worker type actually fixes More DPU per worker means a bigger executor: more heap for each task's working set, more cores sharing that heap, and more local disk for shuffle and spill. It is the right move when the symptom is **per-executor pressure**: - Executors dying with out-of-memory errors. - Heavy spill to disk during shuffles or sorts. - A broadcast side that is too big to fit comfortably. - Genuine skew where one partition is far larger than the rest and simply needs somewhere to live. It is the wrong move when the symptom is **not enough parallelism** — the job is CPU-bound across many well-balanced tasks and the cluster is simply too small. There, more workers is the answer, and moving to G.2X while halving the count makes it worse. It is also the wrong move for the most common Glue slowness of all: a source split into tens of thousands of tiny objects, where the job spends its time listing and opening files rather than computing. No worker size fixes that; compaction upstream does. ## Auto scaling Setting the job parameter `--enable-auto-scaling` to `true` (available from Glue 3.0 onward) lets Glue add and remove executors during the run based on demand, up to the maximum number of workers you configured. Since you are billed for what is actually consumed, this suits jobs whose shape varies run to run — a backfill that reads a month one day and an hour the next, or a pipeline with a heavy stage followed by a light one. It does not remove the need to pick a sensible worker type: auto scaling changes *how many* executors exist, not *how big* each one is. For jobs with no latency requirement, Glue also offers a **FLEX** execution class, which runs on spare capacity in exchange for less predictable start and run times. It is appropriate for backfills and non-urgent nightly work, and inappropriate for anything with an SLA attached. ## How to size in practice Start from evidence, not from a default: 1. Turn on the job's metrics and the Spark UI (`--enable-metrics`, `--enable-spark-ui`) so you can see executor memory, task duration distribution and shuffle spill instead of guessing. 2. Diagnose the symptom class. OOM and spill → bigger worker type. Uniformly long stages with all executors busy → more workers. Long tail of one or two tasks → skew, which needs a data fix (salting, a different partition key, a broadcast) more than a hardware fix. 3. Change one variable at a time and compare DPU-hours, not wall clock — a job that finishes in half the time on twice the capacity has not saved anything. 4. Re-measure after the data grows. Glue sizing set once during a proof of concept and never revisited is where a large share of a data platform's bill quietly accumulates. ## What interviewers are checking That you understand a worker type is a per-executor size, not a throughput dial; that you can express the cost identity (type × count × duration) and therefore know the neutral substitution; and that you reach for measurement before capacity. The weak answer is "the job was slow so we moved to G.2X" with no account of what was actually constrained.