skip to content

In Django, a feed shows 50 likes and reads like.content_object for each; how do prefetch_related and GenericPrefetch cut the query count?

level: seniorimportance: nice to knowfreq 22%

answer

  1. one query per access
  2. group by content type
  3. one pk__in per model
  4. a queryset per type (5.0)

basics

~20 s

Each content_object read runs its own query. prefetch_related('content_object') groups likes by content type and runs one query per target model; GenericPrefetch, new in Django 5.0, supplies a custom queryset per type, such as one using select_related or only().

solid answer

~30 s

Without help, the feed runs one query for the likes and then one per `like.content_object` read, since content type lookups come from the cache. `prefetch_related('content_object')` collects the likes' `(content_type_id, object_id)` pairs, runs one `pk__in` query per target model, and matches results in Python, so 50 likes over photos, comments and posts cost four queries. `select_related('content_object')` is not an option; it raises `FieldError`. When each target needs its own loading, `GenericPrefetch('content_object', [Photo.objects.select_related('owner'), Comment.objects.select_related('author')])` from `django.contrib.contenttypes.prefetch` supplies one queryset per content type. Passing two querysets for the same model raises `ValueError`, and `values()` or `raw()` querysets are rejected.

code

python · 18 lines
python
from django.contrib.contenttypes.prefetch import GenericPrefetch

likes = (
    Like.objects.filter(user=request.user)
    .prefetch_related(
        GenericPrefetch(
            'content_object',
            [
                Photo.objects.select_related('owner'),
                Comment.objects.select_related('author'),
                Post.objects.only('id', 'title'),
            ],
        )
    )
    .order_by('-created_at')[:50]
)
for like in likes:          # 1 query for likes + 1 per content type
    print(like.content_object)

go deeper

for a junior

Recall that reading content_object in a loop costs a query per row, and that prefetch_related helps where select_related cannot.

for a middle

Explain the grouping by content type and the one-query-per-model result, and count the queries for a mixed feed.

for a senior

Show you would reach for GenericPrefetch to shape each type's query, and know its ValueError rules and how it pairs with slicing.

for a principal

Consider whether a hot feed should read through generic links at all, or through a denormalised feed table that avoids per-type fan-out.

## Where the queries come from Reading `like.content_object` on a Like that has not been prefetched does two things: it resolves the content type through `ContentType.objects.get_for_id()`, which is served from the in-process cache after the first time, and then it runs a `get()` on the target model for `pk=object_id`. In a loop over 50 likes this is the classic **N+1**: one query for the list, fifty for the targets. With Django 6.1's default fetch mode, `FETCH_ONE`, each access fetches only for the current instance. ## How prefetch_related handles a GenericForeignKey `prefetch_related('content_object')` has special support for generic foreign keys: 1. Evaluate the likes query. 2. Group the likes by `content_type_id`, collecting their `object_id` values; likes with a null type or id are skipped. 3. For each content type, run one query: `Model._base_manager.filter(pk__in=ids)`. 4. Match the results to the likes in Python on the pair (model class, primary key as a string) and cache each target on its like. For 50 likes spread over photos, comments and posts, that is **four queries**: one for likes and one per target model. The number grows with the number of distinct content types, not the number of likes. `select_related('content_object')` cannot do this in SQL; it raises `FieldError` because there is no single table to join. ## GenericPrefetch: a queryset per content type Plain prefetching uses each model's base manager with no options. A feed usually needs more: the photo's owner, the comment's author, only a few columns of a post. `Prefetch` accepts one queryset, which cannot fit a lookup whose targets are several models. Django 5.0 added `GenericPrefetch(lookup, querysets, to_attr=None)` in `django.contrib.contenttypes.prefetch`: - `querysets` is a list with **one queryset per content type**; Django picks the queryset whose model matches each group and adds the `pk__in` filter itself. - A content type without a matching queryset falls back to the default behaviour. - Two querysets for the same model raise `ValueError` ("Only one queryset is allowed for each content type") when the prefetch runs. - Querysets built with `values()`, `values_list()` or `raw()` are rejected with `ValueError` when the `GenericPrefetch` is constructed, because the results must be model instances. - `to_attr` stores the result under another attribute name, as with `Prefetch`. | Approach | Queries for 50 likes over 3 types | Per-type options | |---|---|---| | no prefetch | 1 + up to 50 | none | | `prefetch_related('content_object')` | 4 | none | | `GenericPrefetch` with `select_related` per type | 4 | yes | | `select_related('content_object')` | `FieldError` | not applicable | ## Verifying the query count Claims about query counts should be checked, not assumed. In a test, wrap the feed code in `self.assertNumQueries(4)` from Django's `TestCase`; in development, count the queries the page issues with a query log or a debugging toolbar. Two details change the count: - the first request in a process may add content type lookups before the cache is warm; - every nested `select_related` stays inside its per-type query, but a nested `prefetch_related` in a per-type queryset adds its own query per type. A regression test on the count is the cheapest way to stop a later change, such as a template that reads `like.content_object.owner` without the matching `select_related`, from quietly reintroducing the N+1. ## Going the other direction Prefetching **from** targets **to** their likes is ordinary: with `likes = GenericRelation(Like)` on Photo, `Photo.objects.prefetch_related('likes')` runs one query for all the photos' likes, filtered by content type and `object_id__in`. That path is a normal reverse relation and needs no `GenericPrefetch`. ## Pitfalls - **Filtering after prefetch.** `like.content_object` is cached, but calling a queryset method on a target's own relations still queries; nest `select_related` in the per-type queryset instead. - **Deleted targets.** A like whose target is gone simply ends up with `None` cached; templates must handle it. - **Slicing order.** Build the queryset with `prefetch_related` and then slice; the prefetch runs once for the sliced page. - **Fetch modes (6.1).** `QuerySet.fetch_mode(models.FETCH_PEERS)` also covers generic relations: the first `content_object` read fetches the targets for every like loaded with it. It is a safety net; explicit `GenericPrefetch` remains the way to shape each type's query.

  • Why can't you pass a single Prefetch('content_object', queryset=Photo.objects.select_related('owner')) for this feed?
    A `Prefetch` takes one queryset, but the targets of `content_object` belong to several models; a Photo queryset cannot load comments or posts. `GenericPrefetch` exists for exactly that case: it takes a list with one queryset per content type and matches each group of likes to the queryset of the right model.
  • How many queries does Photo.objects.prefetch_related('likes') run when Photo has likes = GenericRelation(Like)?
    Two, beyond content type lookups that come from the cache: one for the photos and one for all their likes, filtered on Photo's content type and `object_id__in` the photo keys. Going from targets to the generic model is a normal reverse relation, so plain `prefetch_related` is enough.

saying these in an interview costs you the question

  • select_related('content_object') loads all targets in one JOIN
  • prefetch_related on a generic key runs one query per like
  • GenericPrefetch accepts values() querysets to save memory
  • Passing two Photo querysets makes GenericPrefetch merge them
  • GenericPrefetch has existed since the contenttypes framework began