In Python, how do you deduplicate a list of records by one field, keeping the first occurrence?

## The problem Order-preserving deduplication has an easy form and a harder one. The easy form collapses elements that are equal to each other, and `list(dict.fromkeys(items))` solves it. The harder form collapses elements that are equal **in one derived value** while keeping the whole element. Deduplicating a replayed stream of scoring events for a fraud-scoring service by `request_id`, keeping the score that was computed first, is the harder form, and no built-in expresses it. The reason is that `dict.fromkeys` hashes the element itself. Two event dicts with the same `request_id` but different `score` are not equal, so they do not collapse. `set` has the same limitation, plus the elements would have to be hashable to begin with. ## The idiom that is correct ```python seen = set() first_only = [] for event in events: rid = event["request_id"] if rid not in seen: seen.add(rid) first_only.append(event) ``` One pass, one hash lookup per element, output built in input order, first record per key. The `seen` set holds only the keys, not the records, so its memory scales with the number of distinct keys and with the size of a key rather than the size of a payload. Write it as a small helper taking a key function and it composes with everything: ```python def dedupe_by(items, key): seen = set() for item in items: k = key(item) if k not in seen: seen.add(k) yield item ``` Generatorising it means the caller decides whether to materialize, and a caller that only needs the first ten unique records stops after ten. ## The one-liner that quietly does something else The comprehension people reach for first is ```python list({e["request_id"]: e for e in events}.values()) ``` It deduplicates, and it even preserves first-seen key order, so it passes a quick eyeball. But it keeps the **last** record for each key, because assigning to an existing key updates the value while leaving the key where it was. Order comes from the first occurrence; content comes from the last. On a replayed stream where the first score is the authoritative one and later replays carry a partially recomputed score, this is a data-corruption bug that no type checker sees. ```pycon >>> events = [{"id": "r1", "score": 0.9}, {"id": "r2", "score": 0.4}, {"id": "r1", "score": 0.1}] >>> {e["id"]: e for e in events} {'r1': {'id': 'r1', 'score': 0.1}, 'r2': {'id': 'r2', 'score': 0.4}} ``` ## Why reversing is not the fix The usual patch is to build the comprehension over `reversed(events)`, so the last assignment for each key is the earliest record. The values are then right, but the ordering is not: keys now enter the dict in order of each key's **last** occurrence, reversed, which is not the same as first-seen order. For events with ids `a, b, a, b`, the reversed comprehension yields keys `b, a` while first-seen order is `a, b`. Reversing the output does not repair it either, since the two orders disagree in different directions for different inputs. If ordering is part of the contract, do not derive it from a dict built backwards; use the `seen` loop, where the ordering is the loop's own. ## Choosing the key The key must be hashable and must capture exactly the notion of sameness you mean. A single id field is the common case. A composite key is a tuple, `(event["tenant"], event["request_id"])`, which is hashable as long as its parts are. A normalized key is often what you actually want, and getting the normalization boundary wrong is its own bug class: deduplicating email addresses on the raw string keeps `[email protected]` and `[email protected]` as two records, while deduplicating on `s.casefold()` merges them. Decide deliberately, and put the decision in the key function where a reader can see it. ## Cost and scale One pass, average O(1) per lookup, so O(n) overall, with memory proportional to the number of distinct keys. At a 1,200-request-per-minute peak, a batch covering an hour is 72,000 events and at most 72,000 short string keys, which is nothing; the payloads would have been the expensive part, and the `seen` set never holds them. The trap at that scale is not CPU, it is a `seen` set that lives in a long-running process and is never trimmed. ## Interview framing The question separates candidates who know a trick from candidates who know what the trick does. Anyone can produce the dict comprehension. The signal is noticing, unprompted, that it keeps the last record and that the requirement said first.