Choosing a Processing Engine

Before the first read, a cluster must be granted capacity, launch a worker process on each machine, copy the program and its libraries to each, build a plan and list the input. None of that touches data.

In a workflow scheduler's graph an edge means only 'start after the previous step reported success'; nothing travels along it. Inside one program handed to a cluster execution engine, an edge carries the records themselves from one step to the next.

A cluster execution engine runs your program on machines that are part of your deployment, over storage you point it at, along a plan your code shapes. A managed query service owns all three and shows you none of them.

A fixed reporting cadence usually means the runtime executes repeated small finite runs: it collects whatever arrived in an interval, runs an ordinary finite job over that slice, and repeats. The floor is the interval plus that run.

Usually no. Bytes on disk are not the test; the working set is. Per-customer totals hold one running number per customer, so the records can be read in passes and discarded, and only the totals must fit in memory.

It has to be redefined. With no last record, no ranking is ever settled, so you either rank within a boundary the job imposes, or publish a running top-10 that is correct only as of the records already seen.

Only the distributed run encodes each value into bytes and decodes it again at every process boundary, adds framing and bookkeeping per unit of work, and ships the program and the values its functions captured to every machine.

Volume is the weakest predictor. What decides is expressibility: whether the work needs arbitrary code and libraries running next to the data, whether the service can read the inputs at all, and where the data already lives.

Find where the seven hours go before adding machines. A quadratic scan, repeated re-reads, and fields nobody uses stay expensive after distribution — and a run pinned to one core on a sixteen-core machine has headroom where it is.

Reconciliation decides which of the two answers a reader sees for each period, at what moment one supersedes the other, whether a published number is allowed to change, and how a disagreement is detected rather than reported by a stakeholder.

Boundedness is whether the record set has a last member, not how the records travel. A watched location that keeps filling is unbounded, and a slice of a never-ending feed pinned between two chosen boundaries is bounded.

A full write and a full read of 400 GB, plus a second program's start-up and a path someone must name, encode and eventually delete. An edge between two steps a scheduler starts carries nothing, so whatever crosses it has to be materialised.

Fixed per-step costs dominate: 40,000 program launches and 40,000 rows of scheduler bookkeeping for a few seconds of real work each, throttled by whatever concurrency limit the scheduler enforces. Fan-out over data belongs inside one submission, as parallelism.

The execution shape fixes the unit of redo. Under repeated small finite runs, at most one interval's slice is re-executed. Under a runtime handling each record on arrival, the redo reaches back to the position the job last saved.

Say whether the number is final or provisional. Over an input with no end a value is true only as of the records seen, so either freeze a bounded slice and publish once, or publish revisions with an as-of marker.

Partial failure: some work is durable and some died with its machine. The team now owns a write path and a rerun that stay correct when the same records arrive twice, monitoring that can tell degraded from failed, and a definition of done that is not an exit code.

Check what the program's own code carries beyond the join and the aggregate — null rules, deduplication, an outside lookup, precision handling — and whether anything needs a library. If nothing does, the extraction and return trip are pure cost.

Computations whose retained state grows with the input — a global sort, an exact distinct count at high cardinality, an exact median — resist bounded memory. One machine still finishes them by writing sorted chunks to local disk and merging them back.

Capacity billed while idle, plus a standing surface: version consistency across every worker, coordinated upgrades, an access and credential model, on-call for infrastructure nobody on the team wrote, and the expertise to run it. That surface is per cluster, so halving the machines does not halve it.

Promote a boundary only when something other than the next step needs it: a different program or runtime, an intermediate real consumers read, an external wait, or two halves wanting very different capacity. Each promotion costs a materialised named artefact, a second start-up, and a second picture to read.

Set the boundary as a written rule, not per pipeline. Default to whichever side already holds the data and the people, and make the escape hatch a capability test rather than a preference. Owning both sides costs a second skill set, a second deploy path and logic that drifts.

Default to one implementation serving both the live run and the recomputation, and make a second implementation a reviewed, time-boxed exception with a named owner. Drift and reconciliation are paid every month; a single path pays mostly up front.

Decide on growth rate, not today's size: track projected months until the working set or the run time reaches what the largest available machine can do. Committing early pays a per-run floor cost; committing late migrates under a failing deadline.