When does a collections.defaultdict accumulation beat itertools.groupby for grouping?

## The real comparison The choice is not "iterator tool versus dict tool". It is: **who guarantees that equal keys are adjacent, and what does that guarantee cost?** `itertools.groupby` assumes adjacency and gives you, in exchange, a single pass in constant extra memory that emits each group the moment it is complete. `collections.defaultdict(list)` assumes nothing about order and gives you, in exchange, a complete mapping — but only after it has read every record, and with every record held in memory. If adjacency is already true of your input, groupby is free and the dictionary is wasteful. If it is not, you must buy adjacency with a sort, and then almost every advantage groupby had is gone. ## What the sort actually costs ```python from collections import defaultdict from itertools import groupby from operator import itemgetter pending = [("ann", "build failed"), ("bob", "review requested"), ("ann", "deploy done")] # one linear pass, no ordering assumption digests = defaultdict(list) for recipient, line in pending: digests[recipient].append(line) # the same result, paid for with a sort by_recipient = {k: [line for _, line in g] for k, g in groupby(sorted(pending, key=itemgetter(0)), key=itemgetter(0))} ``` The sorted route is `O(n log n)` comparisons plus a key call per item, and `sorted()` builds a new list of every record. The dictionary route is `O(n)` expected, with one hash per item, and it never holds more than the records themselves. On top of that the second form is a longer expression with a key function repeated twice — a place the classic bug (sort by one key, group by another) can hide. Order of results differs too. Insertion-ordered dicts, guaranteed since Python 3.7, mean the accumulation preserves *first-seen* order of the keys, which is often what a human-facing report wants; the sorted route gives you key order, which is what a diffable artefact wants. Pick deliberately rather than by accident. ## Key constraints point in opposite directions - Dictionary accumulation requires **hashable** keys, and nothing else. A list key fails; an unorderable key is fine. - The sort-then-groupby route requires **orderable** keys, and nothing else. Keys that are unhashable but comparable work; mixed types that do not compare raise `TypeError`. That is a genuine deciding factor, not trivia: it is the reason a grouping over dictionaries or lists as keys usually ends up on the groupby side after a sort by some derived tuple, while a grouping over frozen or scalar keys goes to the dictionary. ## When groupby is still the right answer **The source is already ordered.** A nightly export written in recipient order, a result set ordered by the group column, or several sorted feeds combined with `heapq.merge` all satisfy the invariant for free. **The grouping does not fit in memory.** This is the case a dictionary cannot serve at all. Consider a digest sender that, at the end of each three-week release train, must turn a long file of accumulated notifications into one message per recipient. A `defaultdict(list)` holds every notification of the whole train before it can send the first message. groupby over the file, if the file is written in recipient order, holds one recipient's notifications, sends, and drops them. ```python import io from itertools import groupby export = io.StringIO("ann,build failed\nann,deploy done\nbob,review requested\n") with export as fh: rows = (line.rstrip("\n").split(",", 1) for line in fh) for recipient, group in groupby(rows, key=lambda r: r[0]): print(recipient, [body for _, body in group]) ``` Note the `with`. A streaming pipeline like this holds the reader open for the entire loop, and a generator suspended over an open handle is exactly how a long-lived process ends up leaking descriptors: if you `break` out of the loop, or an exception unwinds it, the generator is left suspended and the file stays open until it is collected. Owning the resource with a context manager rather than trusting the iterator to reach its end is the discipline this shape demands. **Adjacency is the semantics.** If you want runs — consecutive duplicates, contiguous stretches of one status — a dictionary answers a different question entirely. ## When neither should store the members A large share of real grouping tasks want a number per key, not a list per key. Then the argument dissolves: accumulate into `defaultdict(int)`, a `collections.Counter`, or a small dictionary of running values, and peak memory becomes the number of distinct keys instead of the number of records. On a sorted stream, groupby can do the same by consuming each group lazily and keeping only the aggregate. ## How to decide, quickly 1. Is the input already ordered by the group key, or can the producer be asked to order it cheaply? If yes, groupby. 2. Must the result fit in memory as a whole mapping? If no, groupby over an ordered source is the only option that works. 3. Are the keys hashable, and is the input unordered? Then a defaultdict pass, and do not add a sort to justify a tool. 4. Do you need members at all, or just counts and sums? If just aggregates, accumulate them directly.