How do you deduplicate a list of unhashable items such as dicts or lists in Python?

## Why the one-liners fail `set(items)` and `dict.fromkeys(items)` both hash every element, so both fail the moment an element is a `dict`, a `list` or a `set`. On CPython 3.14 the error is explicit about what was being attempted: ```pycon >>> set([{"a": 1}]) TypeError: cannot use 'dict' as a set element (unhashable type: 'dict') >>> dict.fromkeys([{"a": 1}]) TypeError: cannot use 'dict' as a dict key (unhashable type: 'dict') ``` Older releases printed only the `unhashable type: 'dict'` half; 3.14 names the container and the role, which is a small but real debugging improvement when the failing expression is a nested comprehension. ## The pattern: canonicalize, then deduplicate The fix is not to make the items hashable, it is to derive a hashable **stand-in** and deduplicate on that while keeping the original objects: ```python seen = set() unique = [] for item in items: key = canonical(item) if key not in seen: seen.add(key) unique.append(item) ``` Everything interesting is in `canonical`, and it is a decision about meaning, not a formality: the key must be equal for exactly the pairs of items you consider duplicates, and different otherwise. ## Choosing a canonical form **A flat list becomes a tuple.** `tuple([1, 2])` is hashable and compares element-wise, so it is the exact hashable mirror of the list. **A flat dict with hashable values has two options.** `frozenset(d.items())` is hashable and, because a frozenset is unordered, treats `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` as the same key, which is usually what you mean for a mapping. `tuple(sorted(d.items()))` gives the same insensitivity to key order plus a deterministic, printable, comparable key, at the cost of requiring the keys to be mutually orderable. Both break if any value is itself a list or dict. **Nested structures need recursion or serialization.** A small recursive canonicalizer turns lists into tuples and dicts into sorted tuples of pairs, all the way down, and preserves types. The blunter option is `json.dumps(obj, sort_keys=True)`, one line and robust against arbitrary nesting, with three real caveats: it only handles JSON-representable data, it erases the tuple/list distinction so `(1, 2)` and `[1, 2]` become the same key, and it is markedly slower per item than hashing a tuple. For deduplicating parsed JSON payloads, which is where this problem usually arrives from, those caveats are harmless and the one-liner wins. **Sometimes the canonical form is a projection, not a faithful encoding.** If two records are duplicates when their id and timestamp match, regardless of every other field, then the key is just that pair, and no encoding of the whole object is needed. Reaching for a full serialization when a two-field tuple would do costs both CPU and clarity. ## Getting sameness wrong is the real bug The canonical function silently defines your notion of equality, and its failure mode is quiet. A key built from `tuple(d.items())` without sorting makes two dicts with the same pairs in a different insertion order count as distinct, so duplicates survive. A key built from `json.dumps` without `sort_keys=True` has the same defect. Conversely, a key that drops a field that actually distinguishes records merges rows that should have stayed separate, and nothing raises. Test the canonicalizer directly with a pair you know should collapse and a pair you know should not; it is a pure function and trivially testable. ## When there is no canonical key For objects with a meaningful `__eq__` but no hash, and no derivable key, the honest fallback is the quadratic scan: ```python unique = [] for item in items: if not any(item == kept for kept in unique): unique.append(item) ``` It preserves first-seen order and works with equality alone, but it costs O(n squared) comparisons and each comparison may itself be deep. Reserve it for tens of items. If you own the class, the better move is to define `__hash__` alongside `__eq__` on an immutable value type, or make it a `frozen=True` dataclass so both are generated and the plain `dict.fromkeys` idiom works again. ## Interview framing The visible skill is knowing that a `TypeError` about hashability means "derive a key", not "give up on sets". The deeper signal is treating the canonical function as the specification of duplicate-ness, naming its failure modes, and choosing the cheapest key that is still faithful.