How do you collect an async generator's values into a list in Python?

middlenice to knowfreq 30%

answer

The builtin container refuses it
One keyword moves into the comprehension
Only legal inside an asynchronous function
Parentheses give you another stream, not a container
[item async for item in stream()]

basics

~20 s

With an async comprehension, [item async for item in stream()], written inside an async def. list() and sorted() raise TypeError on an async generator because they need __iter__, and the standard library has no helper that materializes one.

solid answer

~50 s

PEP 530 (Python 3.6) allows an async for clause inside list, set and dict comprehensions and inside generator expressions, and allows await in the element expression or the condition. So [item async for item in stream()] drains an async generator into a list. The whole comprehension has to sit inside an async def body — at module level it is a SyntaxError about an asynchronous comprehension outside an asynchronous function — although the asyncio REPL, python -m asyncio, accepts one at the prompt. Two traps: a parenthesized (item async for item in stream()) is not a container but another async generator object, so it must itself be driven with async for; and list(), sorted() and friends refuse an async iterable outright. Materializing also throws away the streaming property, so for an unbounded feed keep iterating or take the first N with anext().

PEP 530 (Python 3.6) allows an `async for` clause inside list, set and dict comprehensions and inside generator expressions, and allows `await` in the element expression or the condition. So `[item async for item in stream()]` drains an async generator into a list. The whole comprehension has to sit inside an `async def` body — at module level it is a `SyntaxError` about an asynchronous comprehension outside an asynchronous function — although the asyncio REPL, `python -m asyncio`, accepts one at the prompt. Two traps: a parenthesized `(item async for item in stream())` is not a container but another async generator object, so it must itself be driven with `async for`; and `list()`, `sorted()` and friends refuse an async iterable outright. Materializing also throws away the streaming property, so for an unbounded feed keep iterating or take the first N with `anext()`.

code

python · 21 lines

python

import asyncio

async def ticks(count):
    for i in range(count):
        await asyncio.sleep(0)
        yield i

async def take(source, limit):
    collected = []
    async for item in source:
        collected.append(item)
        if len(collected) == limit:
            break
    return collected

async def main():
    print([i async for i in ticks(3)])
    print(sorted({i async for i in ticks(6) if i % 2 == 0}))
    print(await take(ticks(1000), 4))

asyncio.run(main())

go deeper

for a junior

Remember the shape [item async for item in stream()] and that it only works inside an async def. Knowing that list() cannot drain an async generator saves a confusing TypeError.

for a middle

Explain why the builtin containers refuse an async iterable — they drive __iter__, whose values cannot come from an await — and know that parentheses produce another async generator rather than a container.

for a senior

Weigh the cost out loud: materializing trades away memory bounds, early-item latency and the ability to stop early, so make the call per stream rather than by habit, and reach for a bounded take when the size is not yours to control.

for a principal

Set the expectation in review: streaming APIs are published so consumers can process incrementally, and a codebase that collects every stream into a list has paid for machinery it is not using while inheriting its failure modes.

The comprehension form

Python 3.6 added asynchronous comprehensions (PEP 530). The rule is small: a comprehension may use async for in place of for for any of its clauses, and it may await in the element expression or in an if condition. That covers all four comprehension forms.

python

values  = [item async for item in stream()]
uniques = {item async for item in stream()}
indexed = {item: len(item) async for item in stream()}
lazy    = (item async for item in stream())

The first three build a list, a set and a dict by draining the async generator to exhaustion. The fourth does something different, and it is the trap: a parenthesized async comprehension is an async generator expression — it evaluates to an async generator object that has produced nothing yet, and it must itself be consumed with async for or another async comprehension.

Why `list()` refuses

list(stream()) raises TypeError: 'async_generator' object is not iterable. The builtin containers, and every function that consumes an iterable — sorted, sum, min, any, tuple — work through __iter__/__next__, which produce values synchronously. An async generator only offers __aiter__/__anext__, whose values arrive through an await, and there is no way for a synchronous call to await anything. The same is true of sorted(item async for item in stream()): the generator expression there is an async generator, so sorted rejects it. Materialize with a comprehension first, then sort the list.

There is no async itertools in the standard library — no ready-made async islice, chain or groupby — so the small helpers you would reach for get written by hand as async generators.

Where the comprehension is allowed

An async comprehension contains an await, so it needs an asynchronous body around it. Inside an async def (including inside another async generator) it is fine; at module level in a normal script it is a SyntaxError reading asynchronous comprehension outside of an asynchronous function. The asyncio REPL started with python -m asyncio runs each statement inside a coroutine, so the same expression works interactively there. This is the single most common paper-cut with the syntax: code that works inside a function is pasted into a module-level script and stops compiling.

The cost you are accepting

An async generator exists to hand out items as they arrive. A comprehension turns that stream into a container, which means:

Memory. Everything the feed produces is resident at once. Fine for three pages of a flight schedule, not fine for the day's full record set.
Latency. The comprehension does not finish until the last item arrives, so the first item is unusable until the last one is fetched. A consumer that could have started work on page one now waits for page forty.
No early exit. The whole point of a stream is that the consumer may stop; a comprehension always drains.

When those matter, keep the async for and process incrementally, or bound the take: loop with a counter and break, or use await anext(iterator, default) to pull one item without a loop. A bounded take is easy to write as three lines and is the honest alternative to materializing an unbounded stream.

`await` inside a comprehension

The second half of PEP 530 lets you await in the element or the condition:

python

checked = [item async for item in stream() if await is_valid(item)]
resolved = [await fetch(key) for key in keys]

The second line is worth pausing on, because it is a common performance mistake. It is a synchronous for over a list of keys with an await in the element expression, so the awaits happen strictly one after another — a hundred keys is a hundred sequential round trips. Concurrency there is not a comprehension feature; it comes from creating a task per key and gathering the results. The comprehension is a shape for expressing the collection, never a parallel map.

Reading it in review

Three signals are worth flagging when this code appears. list() or sorted() applied to something async — a bug that fails at runtime with a TypeError. A comprehension draining a feed whose size is unbounded, or bounded only by an external system — a memory incident waiting for a busy day. And a materialized list that is immediately looped over once, which is a stream that was collected for no reason and can go straight back to being an async for.

### The comprehension form Python 3.6 added asynchronous comprehensions (PEP 530). The rule is small: a comprehension may use `async for` in place of `for` for any of its clauses, and it may `await` in the element expression or in an `if` condition. That covers all four comprehension forms. ```python values = [item async for item in stream()] uniques = {item async for item in stream()} indexed = {item: len(item) async for item in stream()} lazy = (item async for item in stream()) ``` The first three build a list, a set and a dict by draining the async generator to exhaustion. The fourth does something different, and it is the trap: a parenthesized async comprehension is an **async generator expression** — it evaluates to an async generator object that has produced nothing yet, and it must itself be consumed with `async for` or another async comprehension. ### Why `list()` refuses `list(stream())` raises `TypeError: 'async_generator' object is not iterable`. The builtin containers, and every function that consumes an iterable — `sorted`, `sum`, `min`, `any`, `tuple` — work through `__iter__`/`__next__`, which produce values synchronously. An async generator only offers `__aiter__`/`__anext__`, whose values arrive through an `await`, and there is no way for a synchronous call to await anything. The same is true of `sorted(item async for item in stream())`: the generator expression there is an async generator, so `sorted` rejects it. Materialize with a comprehension first, then sort the list. There is no async `itertools` in the standard library — no ready-made async `islice`, `chain` or `groupby` — so the small helpers you would reach for get written by hand as async generators. ### Where the comprehension is allowed An async comprehension contains an `await`, so it needs an asynchronous body around it. Inside an `async def` (including inside another async generator) it is fine; at module level in a normal script it is a `SyntaxError` reading *asynchronous comprehension outside of an asynchronous function*. The asyncio REPL started with `python -m asyncio` runs each statement inside a coroutine, so the same expression works interactively there. This is the single most common paper-cut with the syntax: code that works inside a function is pasted into a module-level script and stops compiling. ### The cost you are accepting An async generator exists to hand out items as they arrive. A comprehension turns that stream into a container, which means: * **Memory.** Everything the feed produces is resident at once. Fine for three pages of a flight schedule, not fine for the day's full record set. * **Latency.** The comprehension does not finish until the last item arrives, so the first item is unusable until the last one is fetched. A consumer that could have started work on page one now waits for page forty. * **No early exit.** The whole point of a stream is that the consumer may stop; a comprehension always drains. When those matter, keep the `async for` and process incrementally, or bound the take: loop with a counter and `break`, or use `await anext(iterator, default)` to pull one item without a loop. A bounded take is easy to write as three lines and is the honest alternative to materializing an unbounded stream. ### `await` inside a comprehension The second half of PEP 530 lets you await in the element or the condition: ```python checked = [item async for item in stream() if await is_valid(item)] resolved = [await fetch(key) for key in keys] ``` The second line is worth pausing on, because it is a common performance mistake. It is a synchronous `for` over a list of keys with an `await` in the element expression, so the awaits happen strictly one after another — a hundred keys is a hundred sequential round trips. Concurrency there is not a comprehension feature; it comes from creating a task per key and gathering the results. The comprehension is a shape for expressing the collection, never a parallel map. ### Reading it in review Three signals are worth flagging when this code appears. `list()` or `sorted()` applied to something async — a bug that fails at runtime with a `TypeError`. A comprehension draining a feed whose size is unbounded, or bounded only by an external system — a memory incident waiting for a busy day. And a materialized list that is immediately looped over once, which is a stream that was collected for no reason and can go straight back to being an `async for`.

What does `(item async for item in stream())` evaluate to?
An async generator object — an async generator expression, not a container and not a tuple. It has produced nothing at the point of evaluation, and it must be consumed with `async for` or fed to an async comprehension. Passing it to `sorted` or `list` fails for the same reason the original stream does: those functions need a synchronous iterable.
Does putting `await` inside a comprehension run the awaits concurrently?
No. `[await fetch(key) for key in keys]` performs the awaits one after another, so a hundred keys costs a hundred sequential round trips. Concurrency comes from creating a task per item and gathering them, then building the list from the results. The comprehension is a collection shape, not a parallel map.
When would you refuse to materialize the stream at all?
When the size is unbounded or set by an external system, when the consumer can act on early items — collecting delays the first item until the last arrives — or when the consumer may stop early, since a comprehension always drains. In those cases keep the `async for`, or bound the take with a counter and a `break`, or `await anext(iterator, default)` for a single item.

saying these in an interview costs you the question

Calls `list()` or `sorted()` on an async generator
Writes an async comprehension at module level
Thinks `(x async for x in s)` is a tuple or a list
Materializes an unbounded stream to sort it
Assumes `await` in a comprehension runs the awaits concurrently

How do you collect an async generator's values into a list in Python?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The comprehension form

Why list() refuses

Where the comprehension is allowed

The cost you are accepting

await inside a comprehension

Reading it in review

▾common follow-ups3 questions

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics4 topics

The comprehension form

Why list() refuses

Where the comprehension is allowed

The cost you are accepting

await inside a comprehension

Reading it in review

answer

go deeper

Why `list()` refuses

`await` inside a comprehension

context

Why `list()` refuses

`await` inside a comprehension