Processing Engines

In spark-submit, --master names the cluster manager and its address (local[*], yarn, k8s://https://host:6443, spark://host:7077), --class is the fully-qualified class holding main() inside the jar, and the trailing application jar is the code Spark ships to the cluster.

In Spark, an action submits a job; the driver cuts that job into stages at shuffle boundaries; each stage runs one task per partition. Tasks are the smallest unit executors actually execute.

With MEMORY_ONLY, a partition that does not fit in storage memory is simply not cached and is recomputed from lineage the next time it is needed. With MEMORY_AND_DISK, that partition is written to the executor's local disk and read back instead of recomputed.

A Transformer implements transform() and converts one DataFrame into another, usually by appending columns. An Estimator implements fit(), which learns from a DataFrame and returns a Model — and that Model is itself a Transformer.

A partition is Spark's unit of parallelism: one partition is processed by one task on one core. The starting count comes from the input — file splits packed to spark.sql.files.maxPartitionBytes (128 MB by default) — not from a fixed number.

Master nodes run the coordinators: the HDFS NameNode (plus JournalNodes and ZKFC when HA is on) and the YARN ResourceManager. Every worker runs a DataNode for storage and a NodeManager for compute, co-located so containers read local blocks.

Dropping a managed table removes its metastore entry and deletes its data files. Dropping an external table removes only the metastore entry and leaves the files untouched. Declare EXTERNAL whenever another system owns the data.

The NameNode keeps the filesystem namespace in memory: directories, files, permissions, and which blocks make up each file. DataNodes store the actual block bytes on local disks. File data never flows through the NameNode.

A Hadoop MapReduce job runs map, then shuffle and sort, then reduce. One map task handles each InputSplit; its output is partitioned and sorted on local disk, fetched by reducers over the network, merged by key, and written out by the OutputFormat.

YARN splits cluster management three ways: the ResourceManager schedules containers across the whole cluster, a NodeManager runs and monitors containers on each machine, and every application gets its own ApplicationMaster that requests containers and drives that job.

HDFS is a hierarchical filesystem where directory rename is an atomic metadata operation, so committing output is nearly free. S3 is a flat key store with no rename: a commit copies every object, so Spark needs a dedicated committer.

Rarely, and only on-premises: an air-gapped or regulated cluster, or a latency-sensitive workload like HBase where compute sits on the same disks as the data. New cloud platforms default to object storage plus an open table format.

YARN still wins on an existing Hadoop estate: data already in HDFS on the same machines, Kerberos and queue policies already configured, and no container images to build or maintain. Greenfield platforms reading object storage normally choose Kubernetes.

Rank jobs by cost and change rate, not by age. Multi-stage pipelines that materialize to HDFS between jobs gain most from Spark on the same YARN cluster; stable, cheap, rarely-touched jobs can stay on MapReduce indefinitely.

Checkpointing makes Flink periodically snapshot every operator's state and every source's read position to durable storage. On failure the job restarts from the last completed snapshot and replays from there, so accumulated state survives without being double-counted.

keyBy() turns a DataStream into a KeyedStream by hash-partitioning records so every record sharing a key reaches the same parallel subtask. Keyed state, keyed timers and keyed windows exist only on a KeyedStream.

Flink's JobManager is the coordinator: it accepts submissions, turns a job into a schedule, allocates task slots and triggers checkpoints. TaskManagers are the worker processes that run the operator subtasks and exchange records directly with each other.

Event time is the timestamp carried inside the record, set when the event happened. Processing time is the clock on the machine running the operator. Event time gives reproducible, order-independent results; processing time gives lower latency and no correctness guarantees.

Call TUMBLE(TABLE impressions, DESCRIPTOR(impression_time), INTERVAL '1' MINUTE) in FROM and GROUP BY window_start, window_end, campaign_id. Each window emits one final insert-only row per campaign when it closes, then its state is purged.

The first rows of a file are whatever was written first - one day, one source, one writer's share - so they carry neither the whole input's key distribution nor the rare record shapes that actually break the job.

The per-record rule can: a plain function taking ordinary values and returning ordinary values. It stops being callable once its signature mentions a runtime type, or it reads configuration, storage or the clock itself instead of taking them as arguments.

Sleeping proves only that the machine's wall clock advanced. Instead hand the job records stamped with chosen moments and advance from the test the job's own claim that nothing older will arrive, so groups close on command.

A distributed run promises the right records, not an order, a file layout or bit-identical arithmetic. A line-by-line comparison asserts all three at once, so it fails on arrangement while every value is correct.

A named operator carries its meaning - the engine knows it keeps rows, or groups by a key - so it may reorder, narrow, fuse or skip it. A handed-over body means only 'call this per record', so it runs exactly where it stands.