skip to content

Aggregation Pipeline

The pipeline language for grouping, joining, reshaping and computing over collections on the server. Interviewers ask because anything past a simple lookup becomes an aggregation, and stage ordering decides whether it is fast or fatal.

part ofMongoDBoverview, primer and where to startread it →
on this pageshow

questions

18

What does _id define in a $group stage, and how do you total a field per group?

level: juniorimportance: must knowfreq 85%

answer

  1. one output document per distinct key
  2. the key is an expression, not just a name
  3. null as the key means one bucket
  4. the dollar prefix makes it a field path
  5. { $sum: 1 } counts, { $sum: "$amount" } totals

basics

~10 s

In $group, _id is the grouping-key expression: one output document per distinct _id value. Setting it to null puts everything in one group. Totals come from accumulators, for example revenue: { $sum: "$amount" }.

solid answer

~40 s

`$group` collapses its input into one output document per distinct value of the `_id` expression. `_id: "$customerId"` groups by that field; `_id: { region: "$region", status: "$status" }` groups by a composite key; `_id: null` collapses everything into a single document. Every other field in the stage must be an **accumulator**: `{ $sum: "$amount" }` totals a field, `{ $sum: 1 }` counts documents, and `$avg`, `$min`, `$max`, `$push` and `$addToSet` cover the rest. Fields you do not accumulate are gone from the output — `$group` does not carry the original document forward unless you accumulate `$$ROOT`. Field paths need the dollar prefix: `_id: "region"` groups everything under the literal string. Output order is unspecified, so add an explicit `$sort` after the stage if you need one.

code

javascript · 12 lines
javascript
db.orders.aggregate([
  { $match: { status: "complete" } },
  { $group: {
      _id: "$customerId",
      orders: { $sum: 1 },
      revenue: { $sum: "$amount" },
      avgTicket: { $avg: "$amount" },
      lastOrderAt: { $max: "$createdAt" }
  } },
  { $sort: { revenue: -1 } },
  { $limit: 10 }
])

go deeper

for a junior

Be ready to write a $group from memory: grouping key in _id with a dollar-prefixed field path, and one accumulator per computed field. Know that { $sum: 1 } counts and { $sum: "$amount" } totals.

for a middle

Explain that _id is an arbitrary expression, that a composite key is just a sub-document read back as $_id.field, and that non-accumulated fields disappear. Say why $sum skips missing values.

for a senior

Show judgment about where the $match goes relative to the $group, and about accumulating $$ROOT on large groups. Be able to reshape the flat $group output back into the document your API expects.

for a principal

Own the question of whether a grouped read should run live at all: pre-aggregated rollup documents versus on-demand $group is a cost and freshness tradeoff you should be able to argue either way.

## What $group is for `$group` is the aggregation pipeline's reduction stage. It consumes the stream of documents produced by the previous stage and emits one document per distinct grouping key. Everything else in aggregation — averages per customer, counts per status, daily revenue — is built on it, which is why fluency with `$group` is effectively the entry ticket to any MongoDB data question. ## The _id expression is the grouping key The `_id` field of a `$group` stage is mandatory and is an **expression**, not merely a field name. Whatever value that expression produces for a document decides which bucket the document lands in, and that value becomes the `_id` of the output document. - `_id: "$customerId"` — one output document per distinct customer id. The dollar prefix is a *field path*; without it you are supplying a literal. `_id: "customerId"` is legal and produces exactly one group whose `_id` is the string `"customerId"`, which is a classic beginner bug. - `_id: null` — a single group containing every input document. Use it for whole-collection totals. - `_id: { region: "$region", tier: "$tier" }` — a composite key. The output `_id` is that sub-document, and later stages reference its parts as `"$_id.region"`. - `_id: { $dateTrunc: { date: "$createdAt", unit: "day" } }` — any expression works, so you can bucket a timestamp by day, group by an uppercased field, or group by a computed flag. Grouping keys are compared by value including BSON type, so the integer `1` and the string `"1"` form different groups, and documents that are missing the field group together under `null`. ## Accumulators do the arithmetic Every other field in a `$group` stage must be an accumulator expression. The common ones: - `$sum` — `{ $sum: "$amount" }` totals the field; `{ $sum: 1 }` counts documents in the group. `$sum` ignores documents where the value is missing or non-numeric rather than erroring, and returns `0` when nothing summed. - `$avg` — arithmetic mean, ignoring non-numeric and missing values. A group with no numeric values yields `null`, not `0`. - `$min` / `$max` — extremes, which work on dates and strings too, using BSON comparison order. - `$push` — collects a value from every document into an array, duplicates included. - `$addToSet` — the same, de-duplicated, with no order guarantee. - `$first` / `$last` — the value from the first or last document to reach the group, which is only meaningful when a `$sort` precedes the stage. A field you do not accumulate does not survive the stage. If you need the whole source document, accumulate it explicitly with `{ $push: "$$ROOT" }` or `{ $first: "$$ROOT" }` — `$$ROOT` is the system variable holding the current input document. ## Shape of a real pipeline A typical grouped report filters first, groups, then orders and truncates: ```javascript db.orders.aggregate([ { $match: { status: "complete" } }, { $group: { _id: "$customerId", orders: { $sum: 1 }, revenue: { $sum: "$amount" }, avgTicket: { $avg: "$amount" }, lastOrderAt: { $max: "$createdAt" } } }, { $sort: { revenue: -1 } }, { $limit: 10 } ]) ``` Putting the `$match` before the `$group` matters: it is the only place a filter can shrink the input the group has to reduce, and the only place a predicate is expressed against the original fields. ## Things that surprise people **Output is unordered.** `$group` makes no promise about the order of the documents it emits. If your report needs an order, add `$sort` after the group — that is what the example above does. Relying on the accidental order you observed in a small collection is a bug waiting for a bigger one. **The output shape is flat and new.** Downstream stages see only `_id` plus the accumulated fields. If you want `customerId` as a normal field again, follow with `{ $set: { customerId: "$_id" } }` or project it. **Counting has two spellings.** `{ $sum: 1 }` inside a `$group` counts documents in the group; the separate `$count` stage counts documents in the whole stream and emits a single document. `$sortByCount: "$field"` is shorthand for grouping by a field with `{ $sum: 1 }` and sorting descending. **Missing values are skipped, not zero.** `$sum` and `$avg` quietly ignore documents where the field is absent, so an average over a partially populated field is the average over the populated subset — which is usually what you want, but you should be able to say so out loud.

  • How do you group by two fields at once, and how do later stages read those parts back?
    Make `_id` a document: `_id: { region: "$region", status: "$status" }`. The output `_id` is that sub-document, so downstream stages reference `"$_id.region"` and `"$_id.status"`. If you want them flat again, follow with `{ $set: { region: "$_id.region", status: "$_id.status" } }` and optionally drop `_id`.
  • How do you keep the original documents through a $group?
    Accumulate the whole document with `$$ROOT`, the system variable for the current input document: `{ docs: { $push: "$$ROOT" } }` keeps them all, `{ sample: { $first: "$$ROOT" } }` keeps one. Pushing every document per group can grow the group state substantially on large groups, so prefer accumulating only the fields you actually need.
  • Does $group return its groups in any particular order?
    No. `$group` gives no ordering guarantee for its output, and the order you happen to observe on a small collection can change with data size, index choice or sharding. Add an explicit `$sort` after the stage when the result order matters.

saying these in an interview costs you the question

  • Thinks $group carries all the original fields forward automatically
  • Writes _id: "region" without the dollar prefix and expects grouping
  • Assumes $group emits groups sorted by _id
  • Believes $sum errors when the field is missing on some documents
  • Cannot say what _id: null does

context

open as a page

What does MongoDB's $lookup equality form return for each input document?

level: juniorimportance: must knowfreq 80%

basics

~20 s

$lookup is a left outer join: for each input document it adds an array field, named by as, holding every document from the from collection whose foreignField equals the input's localField. No match yields an empty array, never a dropped document.

open as a page

What does $unwind do to a document whose array field is empty or missing?

level: middleimportance: must knowfreq 72%

basics

~20 s

$unwind emits one document per array element, copying the other fields. A document whose array is empty, missing or null produces no output at all — it is dropped — unless you use the object form with preserveNullAndEmptyArrays: true, which passes it through once.

open as a page

How do you filter or reshape the joined side of a $lookup using let and a sub-pipeline?

level: middleimportance: must knowfreq 62%

basics

~20 s

Use the pipeline form: let binds fields of the input document to variables, and the sub-pipeline runs against the foreign collection with those variables available as $$name. Compare them inside a $match with $expr, then $project or $limit to trim what comes back.

open as a page

Why should $match come early in an aggregation pipeline, and when does MongoDB reorder it for you?

level: middleimportance: must knowfreq 76%

basics

~20 s

Only a leading $match becomes an indexed query on the collection, so filtering first shrinks the stream before expensive stages run. MongoDB's optimizer moves $match ahead of $project, $addFields or $sort only when the filter uses no field those stages create.

open as a page

A $group over 300 million documents spills to disk and runs for an hour — how would you speed it up?

level: seniorimportance: must knowfreq 60%

basics

~20 s

Cut what reaches the $group: an index-backed $match at the head of the pipeline, fewer carried fields, and a coarser grouping key. $group memory scales with distinct keys and accumulator state, so if the key is genuinely huge, split the work by time window and materialize partial results incrementally.

open as a page

What does allowDiskUse: true do in a MongoDB aggregate() call?

level: juniorimportance: should knowfreq 58%

basics

~20 s

allowDiskUse: true lets blocking aggregation stages such as $sort and $group write temporary files on disk when their working set exceeds the 100 MB per-stage memory budget, instead of the pipeline failing with a memory-limit error.

open as a page

How does $addFields differ from $project, and when do you need $project instead?

level: middleimportance: should knowfreq 55%

basics

~20 s

$addFields, and its alias $set, adds or overwrites the fields you name while keeping every other field. $project builds a new shape — an inclusion projection keeps only the fields you list — so use $project when you need to drop or rename fields.

open as a page

What does $expr allow inside a $match stage that ordinary query operators cannot do?

level: middleimportance: should knowfreq 50%

basics

~20 s

$expr embeds aggregation expressions in a query filter, so a $match can compare two fields of the same document — { $expr: { $gt: ["$spent", "$budget"] } } — or apply expression operators such as $dateTrunc, which plain query operators cannot express.

open as a page

What does the $unionWith stage add to a pipeline, and how does it differ from $lookup?

level: middleimportance: should knowfreq 30%

basics

~20 s

$unionWith appends documents from a second collection into the current pipeline's stream, optionally transformed by a sub-pipeline first. Unlike $lookup, which attaches matches as an array field on existing documents, it concatenates rather than joins, and it does not remove duplicates.

open as a page

Why does $unwind after $lookup drop documents, and how do you keep them?

level: middleimportance: should knowfreq 52%

basics

~20 s

Unwinding an empty array produces no output document, so input documents that matched nothing disappear and the left outer join becomes an inner join. Pass preserveNullAndEmptyArrays: true to keep them; the unwound field is simply absent on those documents.

open as a page

How do $out and $merge differ when writing aggregation results to a collection?

level: middleimportance: should knowfreq 50%

basics

~20 s

$out replaces a target collection wholesale with the pipeline's results, keeping the collection's existing indexes. $merge writes incrementally into an existing collection, matching on the on field and applying whenMatched and whenNotMatched rules, so it can update in place rather than rebuilding.

open as a page

When can a $sort in an aggregation pipeline use an index, and what does a following $limit change?

level: middleimportance: should knowfreq 52%

basics

~20 s

A $sort can use an index only while the pipeline is still reading collection documents — at the head, optionally after a leading $match the same index supports. A $limit right after $sort is folded into it, so only the top n documents are held instead of the whole input.

open as a page

How does $facet let one pipeline compute several independent aggregations over the same input?

level: seniorimportance: should knowfreq 38%

basics

~20 s

$facet runs several named sub-pipelines over the same input documents in one pass and returns a single document with one array per facet — typically a page of results, a total count and category buckets in one round trip.

open as a page

How do you return the top 3 highest-scoring documents per group in an aggregation pipeline?

level: seniorimportance: should knowfreq 42%

basics

~10 s

Sort descending, then keep only the leaders per group: $sort followed by $group using $topN or $firstN (MongoDB 5.2+), or $push plus $slice on older versions. $setWindowFields with $rank and a $match also works.

open as a page

How does $graphLookup traverse a hierarchy, and what do connectFromField and connectToField do?

level: seniorimportance: should knowfreq 38%

basics

~20 s

$graphLookup starts from the value in startWith, matches it against connectToField in the from collection, then takes connectFromField from each document it finds and repeats. All documents reached are returned flattened into one array named by as, with recursion capped by maxDepth.

open as a page

What does explain() on an aggregation pipeline show that the pipeline text does not?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Explain output shows the optimized pipeline — the stages array after reordering and coalescing — plus a leading $cursor stage holding the query plan for the part the query layer executed. On a sharded cluster it also shows how the pipeline was split between shards and the merging node.

open as a page

How would you decide whether a $lookup-heavy pipeline can scale in a sharded cluster?

level: principalimportance: should knowfreq 34%

basics

~20 s

Judge it by how many times the join runs and how well each run is targeted: the foreign side is looked up per input document, so shrink the left side first, index the foreign field, and check whether lookups can be routed to one shard. If they cannot, materialize instead.

open as a page