Why does unittest's assertCountEqual pass where assertEqual fails on two lists?

middleshould knowfreq 40%

answer

The name says counts, the meaning is bags
Order is not part of the contract
Compare with sorted() and see what breaks
Duplicates survive here, not in a set
Needs __eq__, not __lt__ or __hash__

basics

~20 s

assertCountEqual compares two iterables as multisets: same elements with the same multiplicities, order ignored. assertEqual on lists uses list equality, which is positional, so a reordered list fails even though both hold exactly the same items.

solid answer

~40 s

TestCase.assertCountEqual(first, second) asserts multiset equality — the same elements in the same quantities, in any order. assertEqual on two lists delegates to list.__eq__, which compares position by position, so ["a", "b"] and ["b", "a"] are unequal. Use the count form when order genuinely is not part of the contract: results gathered from concurrent workers, ids collected out of a set, anything whose sequence is an implementation accident. It is better than assertEqual(sorted(a), sorted(b)) because it needs only __eq__ — lists of dicts, or of mixed types, cannot be sorted at all — and because its failure message reports which elements are out of balance rather than dumping two long lists. It is not assertSetEqual, which discards duplicate counts.

`TestCase.assertCountEqual(first, second)` asserts multiset equality — the same elements in the same quantities, in any order. `assertEqual` on two lists delegates to `list.__eq__`, which compares position by position, so `["a", "b"]` and `["b", "a"]` are unequal. Use the count form when order genuinely is not part of the contract: results gathered from concurrent workers, ids collected out of a set, anything whose sequence is an implementation accident. It is better than `assertEqual(sorted(a), sorted(b))` because it needs only `__eq__` — lists of dicts, or of mixed types, cannot be sorted at all — and because its failure message reports which elements are out of balance rather than dumping two long lists. It is not `assertSetEqual`, which discards duplicate counts.

code

python · 9 lines

python

import unittest

tc = unittest.TestCase()
tc.assertCountEqual([{"id": 2}, {"id": 1}], [{"id": 1}, {"id": 2}])
tc.assertSetEqual({1, 1, 2}, {1, 2})
try:
    tc.assertCountEqual([1, 1, 2], [1, 2, 2])
except AssertionError as exc:
    print(str(exc).splitlines()[0])

go deeper

for a junior

Recall that list equality is positional, so two lists with the same items in a different order are not equal, and that unittest has a dedicated assertion for the order-insensitive case.

for a middle

Explain multiset semantics precisely, including that duplicates count, and be able to say why sorting both sides is a weaker substitute that breaks on dicts and mixed types.

for a senior

Show judgement about which contract the test states. Reaching for an order-insensitive assertion to stop a flaky failure deletes a guarantee; make that a deliberate, documented decision rather than a quick fix.

for a principal

Own the convention that a test asserts the contract and not the current implementation. Where output order is an accident of concurrency, the API documentation and the assertions should agree that it is unspecified.

The name is the worst thing about the method

TestCase.assertCountEqual(first, second) sounds like it compares lengths. It does not. It asserts that the two iterables contain the same elements with the same multiplicities, in any order — multiset (bag) equality. [1, 1, 2] and [2, 1, 1] are count-equal; [1, 1, 2] and [1, 2, 2] are not, even though both have three elements and the same distinct values.

assertEqual on two lists is a completely different assertion: list.__eq__ is positional, so it compares element 0 to element 0 and stops at the first mismatch. A reordered list fails, and the assertListEqual diff shows the two sequences.

When the distinction matters

Whenever the code under test produces a collection whose order is not part of its contract. Consider a chat-transcript archiver that fans a batch out across several shards and gathers the archived message ids back as they complete: the set of ids is a guarantee, the arrival order is an implementation accident. Asserting the order with assertEqual buys you a test that fails on a scheduling hiccup and teaches a four-person team to re-run the suite until it is green — the worst possible lesson. assertCountEqual states the real contract.

The inverse mistake is worse. If the archiver does promise chronological order, reaching for assertCountEqual because the ordered assertion was flaky deletes the guarantee from the test suite without deleting it from the docs. Order-insensitivity is a statement about the contract, not a way to quiet a failure.

Why not `assertEqual(sorted(a), sorted(b))`

That is the reflex answer, and it breaks on real data.

Unorderable elements. Sorting a list of dict objects raises TypeError: '<' not supported between instances of 'dict' and 'dict'. Rows, records and parsed payloads are usually dicts.
Mixed types. A list holding both int and str cannot be sorted at all in Python 3.
Unhashable is fine, unsortable is fatal. assertCountEqual has a fast path that counts hashable elements, and falls back to a quadratic pairwise-matching pass when they are unhashable. It needs __eq__, not __lt__ and not __hash__.
The message. A failed sorted comparison dumps two long lists at you. assertCountEqual prints Element counts were not equal: followed by First has N, Second has M: <element> for each element that is out of balance — usually one or two lines that name the actual defect.

Neighbouring assertions it is confused with

assertSetEqual(a, b) (and comparing set(a) == set(b) by hand) discards duplicates: {1, 1, 2} is just {1, 2}, so a bug that emits a message twice sails through a set comparison and is caught by assertCountEqual. Use the set form only when de-duplication is genuinely part of the contract.

assertIn(item, container) is a membership check on one element and delegates to in (__contains__), so it is the right tool for "the archive contains this id" and the wrong tool for "the archive contains exactly these ids" — a loop of assertIn calls will never catch an extra element.

Two practical details

Both arguments may be any iterable, including a one-shot generator: the method materialises them internally. This matters because assertEqual(list_of_ids, (i for i in ids)) is always a failure — a list is never equal to a generator object, whatever it yields — and that is a genuinely common mis-assertion. assertCountEqual accepts the generator and compares its contents.

Second, elements are compared with ==, so a custom __eq__ participates. If your element type defines equality loosely (say, ignoring a timestamp field), count-equality inherits that looseness, and the assertion is only as strict as the type you are storing.

Reading the failure, and one cost note

A failed count comparison prints Element counts were not equal: followed by a First has N, Second has M: <element> line per unbalanced element — normally the one or two elements that actually differ, not the whole collection. That is the practical reason to prefer it over hand-rolled comparisons even when a hand-rolled one would work: the message points at the defect.

The cost is worth knowing for large collections. The hashable path is linear, but as soon as an element is unhashable the method falls back to matching each element of one sequence against the remaining elements of the other, which is quadratic. For a handful of records that is irrelevant; for tens of thousands of dicts it is a test that takes noticeable seconds, and the fix is usually to compare a projection — a list of ids, or of small tuples — rather than the full records.

### The name is the worst thing about the method `TestCase.assertCountEqual(first, second)` sounds like it compares lengths. It does not. It asserts that the two iterables contain **the same elements with the same multiplicities, in any order** — multiset (bag) equality. `[1, 1, 2]` and `[2, 1, 1]` are count-equal; `[1, 1, 2]` and `[1, 2, 2]` are not, even though both have three elements and the same distinct values. `assertEqual` on two lists is a completely different assertion: `list.__eq__` is positional, so it compares element 0 to element 0 and stops at the first mismatch. A reordered list fails, and the `assertListEqual` diff shows the two sequences. ### When the distinction matters Whenever the code under test produces a collection whose *order is not part of its contract*. Consider a chat-transcript archiver that fans a batch out across several shards and gathers the archived message ids back as they complete: the set of ids is a guarantee, the arrival order is an implementation accident. Asserting the order with `assertEqual` buys you a test that fails on a scheduling hiccup and teaches a four-person team to re-run the suite until it is green — the worst possible lesson. `assertCountEqual` states the real contract. The inverse mistake is worse. If the archiver *does* promise chronological order, reaching for `assertCountEqual` because the ordered assertion was flaky deletes the guarantee from the test suite without deleting it from the docs. Order-insensitivity is a statement about the contract, not a way to quiet a failure. ### Why not `assertEqual(sorted(a), sorted(b))` That is the reflex answer, and it breaks on real data. * **Unorderable elements.** Sorting a list of `dict` objects raises `TypeError: '<' not supported between instances of 'dict' and 'dict'`. Rows, records and parsed payloads are usually dicts. * **Mixed types.** A list holding both `int` and `str` cannot be sorted at all in Python 3. * **Unhashable is fine, unsortable is fatal.** `assertCountEqual` has a fast path that counts hashable elements, and falls back to a quadratic pairwise-matching pass when they are unhashable. It needs `__eq__`, not `__lt__` and not `__hash__`. * **The message.** A failed sorted comparison dumps two long lists at you. `assertCountEqual` prints `Element counts were not equal:` followed by `First has N, Second has M: <element>` for each element that is out of balance — usually one or two lines that name the actual defect. ### Neighbouring assertions it is confused with `assertSetEqual(a, b)` (and comparing `set(a) == set(b)` by hand) **discards duplicates**: `{1, 1, 2}` is just `{1, 2}`, so a bug that emits a message twice sails through a set comparison and is caught by `assertCountEqual`. Use the set form only when de-duplication is genuinely part of the contract. `assertIn(item, container)` is a membership check on one element and delegates to `in` (`__contains__`), so it is the right tool for "the archive contains this id" and the wrong tool for "the archive contains exactly these ids" — a loop of `assertIn` calls will never catch an extra element. ### Two practical details Both arguments may be **any iterable**, including a one-shot generator: the method materialises them internally. This matters because `assertEqual(list_of_ids, (i for i in ids))` is *always* a failure — a list is never equal to a generator object, whatever it yields — and that is a genuinely common mis-assertion. `assertCountEqual` accepts the generator and compares its contents. Second, elements are compared with `==`, so a custom `__eq__` participates. If your element type defines equality loosely (say, ignoring a timestamp field), count-equality inherits that looseness, and the assertion is only as strict as the type you are storing. ### Reading the failure, and one cost note A failed count comparison prints `Element counts were not equal:` followed by a `First has N, Second has M: <element>` line per unbalanced element — normally the one or two elements that actually differ, not the whole collection. That is the practical reason to prefer it over hand-rolled comparisons even when a hand-rolled one would work: the message points at the defect. The cost is worth knowing for large collections. The hashable path is linear, but as soon as an element is unhashable the method falls back to matching each element of one sequence against the remaining elements of the other, which is quadratic. For a handful of records that is irrelevant; for tens of thousands of dicts it is a test that takes noticeable seconds, and the fix is usually to compare a projection — a list of ids, or of small tuples — rather than the full records.

Why not simply assert sorted(first) == sorted(second)?
Because sorting demands an ordering the elements may not have. A list of dicts raises `TypeError: '<' not supported between instances of 'dict' and 'dict'`, and a list mixing `int` and `str` cannot be sorted at all in Python 3. `assertCountEqual` needs only `__eq__`, with a hashable fast path and a quadratic matching fallback. Its message also names the unbalanced elements instead of printing two sorted lists for you to eyeball.
How does assertCountEqual differ from comparing the two collections as sets?
A set comparison discards multiplicity: `{1, 1, 2}` is `{1, 2}`, so a bug that emits the same message twice passes a set assertion and is caught by the count assertion. Use `assertSetEqual` or `set()` only when de-duplication is itself part of the contract; otherwise the count form is strictly stronger.
When is order-insensitive comparison the wrong assertion to reach for?
When ordering is a promise. If an archiver guarantees chronological output, switching to `assertCountEqual` because the ordered assertion went flaky deletes the guarantee from the suite while leaving it in the documentation. Fix the ordering or fix the contract; do not weaken the assertion to silence it.

assertEqual checks two decks of cards are in the same order; assertCountEqual only checks both decks hold the same cards.

saying these in an interview costs you the question

Thinks assertCountEqual only compares the two lengths
Says it ignores duplicate counts, like a set comparison
Claims the elements must be hashable or sortable
Uses it to silence a genuine ordering guarantee
Believes assertEqual on two lists already ignores order
Asserts membership in a loop and calls it an equality check

Why does unittest's assertCountEqual pass where assertEqual fails on two lists?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The name is the worst thing about the method

When the distinction matters

Why not assertEqual(sorted(a), sorted(b))

Neighbouring assertions it is confused with

Two practical details

Reading the failure, and one cost note

▾common follow-ups3 questions

▾analogy~10 s

▾red flags6 flags

§ context

▾related questions7 questions

▾related topics3 topics

The name is the worst thing about the method

When the distinction matters

Why not assertEqual(sorted(a), sorted(b))

Neighbouring assertions it is confused with

Two practical details

Reading the failure, and one cost note

answer

go deeper

Why not `assertEqual(sorted(a), sorted(b))`

context

Why not `assertEqual(sorted(a), sorted(b))`