skip to content

In Redis, when do you use FT.AGGREGATE rather than FT.SEARCH, and what makes an aggregation pipeline expensive?

level: middleimportance: should knowfreq 36%

answer

  1. documents back, or computed rows
  2. same query string, then a pipeline
  3. what forces a read of each hash
  4. sort everything, or only the top

basics

~20 s

FT.SEARCH returns matching documents, ranked or sorted. FT.AGGREGATE takes the same query and runs the matches through GROUPBY, REDUCE, APPLY and SORTBY steps. Pipelines get expensive when they read hashes with LOAD or sort every record.

solid answer

~40 s

I use `FT.SEARCH` when the caller wants documents back: a results page with a total count, 10 items by default. I use `FT.AGGREGATE` when the answer is computed from the matches, such as facet counts per brand or an average price, because it feeds them through ordered `GROUPBY`, `REDUCE`, `APPLY`, `SORTBY` and `FILTER` steps. The query string is the same in both. The cost lies in what the pipeline touches. `LOAD` executes the equivalent of `HMGET` per record, which is slow over millions of keys, so fields I aggregate on are declared `SORTABLE`. `SORTBY` without `MAX` sorts every record, so I add `MAX` for a top-N. For long exports I use `WITHCURSOR` and `FT.CURSOR READ`. I also watch the timeout: the default policy returns partial results, so a count can come back short.

code

plaintext · 5 lines
plaintext
FT.AGGREGATE idx:products "@category:{laptops}" GROUPBY 1 @brand REDUCE COUNT 0 AS n REDUCE AVG 1 @price AS avg_price SORTBY 2 @n DESC MAX 10

FT.AGGREGATE idx:products "*" LOAD 1 @title WITHCURSOR COUNT 500 MAXIDLE 60000
FT.CURSOR READ idx:products 342459320 COUNT 500
FT.CURSOR DEL idx:products 342459320

go deeper

for a junior

Remember that FT.SEARCH returns documents with a total count and 10 rows by default, while FT.AGGREGATE returns rows computed from the matches.

for a middle

Walk through a pipeline in order and name what is expensive: LOAD per record, SORTBY without MAX, and a wildcard start. Say why SORTABLE helps.

for a senior

Bring up the timeout policy and what partial results do to a report, and use FT.PROFILE to find which pipeline step is spending the time before changing anything.

for a principal

Decide which reports may run on the serving instance at all, since a long aggregation competes with search traffic, and set the timeout policy per use rather than by default.

## Two commands, one query language `FT.SEARCH` and `FT.AGGREGATE` both start from the same query string: `@category:{laptops} @price:[500 1500]` means the same thing to each. They differ in what happens to the matches. | | `FT.SEARCH` | `FT.AGGREGATE` | |---|---|---| | Result | Documents, ranked or sorted | Rows computed by a pipeline | | Typical use | A results page | Facet counts, averages, reports | | Default page | `LIMIT 0 10` | Controlled by your `LIMIT` and `SORTBY` steps | | Paging long results | `LIMIT` with a sort | `WITHCURSOR` and `FT.CURSOR READ` | ## What FT.SEARCH replies The reply starts with the **total number of matches**, followed by key and field pairs. That total usually exceeds the rows returned, because `LIMIT` defaults to `0 10`. `LIMIT 0 0` returns only the count. `RETURN` trims the fields, and `NOCONTENT` returns only the keys. The docs warn that `LIMIT` without sorting is **non-deterministic**: pages can repeat or miss documents. For stable paging, add `SORTBY` on a unique field or use `FT.AGGREGATE` with `WITHCURSOR`. `search-max-search-results` caps results when `LIMIT` is used, at 1000000 by default in Redis Open Source. ## Choosing for a catalogue page A typical category page needs both. The product list is an `FT.SEARCH` with a filter, `SORTBY price ASC`, `RETURN` for the fields the page shows, and a `LIMIT` window. The facet sidebar, with its count per brand, is a separate `FT.AGGREGATE` over the same query string with `GROUPBY` and `REDUCE COUNT`. Neither replaces the other: `FT.SEARCH` has no grouping, and an aggregation does not hand back ranked documents unless you load them field by field. ## How a pipeline runs `FT.AGGREGATE` applies its steps in the order you write them, and each works on the output of the one before. A typical pipeline looks like this: 1. **`GROUPBY`** groups rows by one or more properties and takes `REDUCE` functions such as `COUNT`, `SUM`, `AVG` or `COUNT_DISTINCT`. 2. **`APPLY`** computes a new property per row, for example `APPLY "@price * 0.9" AS discounted`. 3. **`SORTBY`** orders the rows, with `nargs` counting its tokens: `SORTBY 2 @n DESC`. 4. **`FILTER`** drops rows using a predicate over the current pipeline state. 5. **`LIMIT`** returns a window of what is left. Because order matters, a `FILTER` on `@n` has to come after the `GROUPBY` that creates it. A `FILTER` step also runs after matching, so a condition the query string can express belongs in the query string. ## Where the cost comes from - **`LOAD`.** Each processed record runs the equivalent of `HMGET` against its key. Over millions of keys that dominates the query. The docs single this out and recommend storing aggregated fields as `SORTABLE`, where the pipeline reads them at very low latency. - **`LOAD *`.** It reads every attribute of every document. The best-practice page lists it as an anti-pattern. - **`SORTBY` without `MAX`.** The pipeline sorts all records and then pages. `SORTBY 1 @foo MAX 100 LIMIT 50 50` sorts only the top 100. - **A broad query.** A wildcard `*` walks all documents. A selective query string shrinks everything downstream. - **`search-max-aggregate-results`.** It is unlimited by default in Redis Open Source, so that cap will not protect you from a careless export. ## Timeouts decide whether a total is true A query that exceeds its timeout follows `search-on-timeout`. The default timeout, `search-timeout`, is 500 ms, and the default policy is `RETURN`, which gives back the results accumulated so far. For a results page that is usually acceptable. For a report, a count that is silently low is worse than an error, so set `FAIL` when correctness matters. ## Cursors for long results `WITHCURSOR` runs the query once and keeps its state on the server, which the docs describe as much quicker than `LIMIT` with an offset. The reply is an array: a batch of rows, then a cursor ID. - **Read size:** `COUNT`, 1000 rows by default, set on the command or on `FT.CURSOR READ`. - **Idle limit:** `MAXIDLE` defaults to 300000 ms and cannot exceed that default; an idle cursor is deleted. - **End of results:** a cursor ID of 0 means nothing remains, and that last reply may still carry rows. - **Clean up:** `FT.CURSOR DEL` frees a cursor you abandon, because `search-index-cursor-limit` allows 128 per shard by default and further attempts fail. - **Freshness:** reads may exclude updates made after the cursor opened.

  • How do you count the matches of a Redis query without paying to fetch any documents?
    Use `FT.SEARCH idx "query" LIMIT 0 0`. The reply carries only the total. That count is accurate unless the query uses `DIALECT 4` sorting optimizations or `WITHOUTCOUNT`, where the query stops once it has its `LIMIT` window; adding `WITHCOUNT` to `SORTBY` restores an accurate figure at the cost of processing every result.
  • Why might an FT.AGGREGATE facet count be lower than the true number with no error returned?
    The query exceeded its timeout, 500 ms by default, and the default `search-on-timeout` policy of `RETURN` handed back what had accumulated. Check the `warning` array in a RESP3 reply and `FT.PROFILE` the query. Then either make it cheaper, raise `TIMEOUT` for that call, or set the policy to `FAIL` so a short answer becomes a visible error.

  • FT.AGGREGATE needs a different query language from FT.SEARCH.
  • FT.SEARCH returns every matching document unless you set a limit.
  • LOAD is free because the fields are already in memory.
  • SORTBY without MAX sorts only as many rows as LIMIT returns.
  • A reply without an error means the aggregate covered all matches.
  • A cursor ID of 0 means the reply it arrived with is empty.