A grouped aggregation over a table is described as three phases — what happens in each?
answer
- one pass, conceptually three parts
- something happens before any computing
- keying, per-key work, reassembly
- the answers still need putting back
basics
~10 sSplit, apply, combine. The split gives every row a grouping key and records which rows share each key; the apply runs one computation per key; the combine assembles those per-key answers into a result.
solid answer
~40 sSplit-apply-combine names one pass in three parts. **Split**: a key expression is evaluated for every row, and rows whose key values compare equal are treated as belonging together — most designs record *which row positions* carry each key rather than copying rows into separate tables. **Apply**: one computation runs per key over that key's rows, either a reduction the library implements itself or a body you supplied. **Combine**: the per-key answers are reassembled into a result, each answer identified by its key value. The phases are a way of reasoning, not a promise of three passes: tools commonly fuse the keying and the computing, and some never assemble a per-key row set at all. Naming them is the entry fee; the discriminating follow-up is which phase your own job is spending its time in.
go deeper
Name the three phases and say what each does in one sentence: rows are given a key, a computation runs once per key, the answers are put back together into a result.
Explain that the split normally produces a record of which rows carry each key rather than a copy, and that tools fuse the phases instead of making three separate passes.
Use the phases as a diagnostic frame: say which one a slow or wrong grouped job is spending its time in, and what evidence pointed you there before you changed anything.
Treat the phase model as shared team vocabulary. A review that can ask which phase a step lives in catches cost and correctness problems that a review of the finished number never sees.
## What the three phases name A **grouped operation** is one pass that gives every row a key, runs a computation once per key, and reassembles the answers into a result. Its conventional name — **split, apply, combine** — describes that pass rather than any one tool's call, which is why it is worth learning as a description: every tabular tool in every ecosystem is documented against it. | phase | what goes in | what comes out | | --- | --- | --- | | split | the table plus a grouping key | a record of which rows carry which key value | | apply | the rows of one key | one answer for that key | | combine | the per-key answers | a result whose rows are identified by key value | ## Split — group identity comes from a key expression The **grouping key** is the column, columns or derived expression whose value decides which rows belong together. The split evaluates it once per row and treats rows whose key values compare equal as one group. - Identity is **equality of the computed value**, not similarity of how the value prints. - What the split normally builds is **bookkeeping** — which row positions carry which key value. Copying each key's rows into a table of its own is something a design may do, but it is not what the phase means. - Tools reach that bookkeeping two ways. A **sorted split** orders the key column so equal keys form a contiguous run. A **hashed split** drops each key value into a bucket by its hash. Both end with the same grouping; they differ in cost and in what else they leave behind. ## Apply — one computation per key The apply runs once per key over that key's rows and returns that key's answer. - A **reduction the library implements itself** — a sum, a count, a minimum — is the fast path: the library knows the operation, so it can fold the values in place without assembling anything per key. - A **hand-written per-group body** is a function you supply that the library cannot look inside. What such a body receives varies sharply between designs: some hand it one record at a time, some hand it the whole group's column at once, and the cost difference between those two is large. ## Combine — the answers become a result The combine puts the per-key answers back together. Two facts are worth having ready: 1. Each answer is identified by its key value. Exactly where those key values land in the result — as row labels or as ordinary columns — is a property of the tool, and only one of those forms is addressable by column name in a later step. 2. The result's row count follows from what the apply returned, not from the table's size. ## The phases are a model, not a schedule Nothing obliges a tool to make three passes, and most do not. - The split and the apply are commonly **fused**: the key is evaluated and that key's accumulator updated in the same sweep over the rows. - Under deferred execution the whole operation may be planned before anything runs, and the phases exist only in the plan. - For a reduction that folds, a tool may carry one accumulator per key and never assemble a per-key row set at all. So the phases are how you *reason* about a grouped operation — where the work is, what could be slow, what could be wrong — not a claim about how many times the data is read. ## What naming the phases does not buy you This is an opening question and interviewers treat it as one. The discrimination is in the follow-ups: - **Which phase costs most here?** It depends on how many distinct keys there are and on what the apply does. - **What order did the result come back in?** Whatever ordering exists is a by-product of the split mechanism, not a guarantee. - **Did anything get copied?** Usually not by the split itself; a copy, when it happens, is demanded by the apply. A candidate who can name the phases and then say which one their own slow job is spending time in is a different candidate from one who can only name them.
- Does a grouped operation necessarily read the data three times, once per phase?No. The phases are a reasoning model. Most tools fuse the keying and the computing into a single sweep, updating each key's accumulator as rows go by. Under deferred execution the phases may exist only in a plan, and a reduction that folds can carry one accumulator per key without ever assembling a per-key row set.
- What decides that two rows belong to the same group?The key expression is evaluated once per row — it may be a column, several columns, or an expression over them — and two rows share a group when those computed values compare equal. Identity is equality of the computed value, not similarity of how the value prints or of the rest of the row.
- Which phase do candidates most often leave out when estimating cost?The split. Because no arithmetic happens there it reads as free, but evaluating the key for every row and then organising the distinct values is frequently the dominant cost of the whole operation, especially when the key is close to unique.
saying these in an interview costs you the question
- Says the split always copies each group's rows into a table of its own.
- Claims the three phases mean three separate passes over the data.
- Thinks the result's row count follows from the table's size, not from the apply.
- Assumes a grouped result always comes back ordered by key.
- Cannot say which phase a slow grouped job is spending its time in.