skip to content

When a Redis sorted set grows past its compact threshold, OBJECT ENCODING reports "skiplist". What two structures does that encoding actually maintain, and why does it need both?

level: middleimportance: should knowfreq 38%

answer

  1. skiplist encoding = skiplist + dict
  2. skiplist: random levels, O(log n), spans give ZRANK
  3. dict: O(1) ZSCORE and ZADD-on-existing
  4. ordered by (score, member)
  5. under 128/64 it is a single listpack blob

basics

~20 s

It keeps a skiplist ordering members by score for O(log n) rank and range queries, plus a hash table mapping member to score for O(1) lookups and updates. Neither structure alone gives both ordered access and constant-time member lookup.

solid answer

~50 s

The `skiplist` encoding is really a **pair** of structures over the same members. - A **skiplist**: a sorted linked list with randomised multi-level express lanes, ordered by (score, member). It gives O(log n) insert, delete, rank and range scans, which is what `ZRANGE`, `ZRANGEBYSCORE`, `ZRANK` and `ZCOUNT` need. - A **hash table** (dict) mapping member → score, giving O(1) `ZSCORE` and O(1) "does this member exist, and what is its current score" during `ZADD`. Both are needed because each answers a question the other cannot answer cheaply. Without the dict, `ZSCORE` would be an O(n) scan and every `ZADD` on an existing member would have to find it by scanning. Without the skiplist, ordered and range queries would require sorting the whole set. The cost is memory: members are referenced from both structures, so a large sorted set is the most expensive core type per element. Below `zset-max-listpack-entries`/`-value` (128/64), Redis instead stores member-score pairs back to back in a single listpack and scans it, which is far smaller.

code

text · 13 lines
text
> ZADD board 100 alice 250 bob
> OBJECT ENCODING board
"listpack"

> ZSCORE board bob          # O(1) once skiplist-encoded (dict lookup)
"250"
> ZRANGE board 0 -1 WITHSCORES   # ordered traversal
> ZRANK board bob           # rank via skiplist spans

# push past zset-max-listpack-entries (128)
> for i in $(seq 1 200); do redis-cli ZADD board $i m$i; done
> OBJECT ENCODING board
"skiplist"

go deeper

for a junior

Know that large sorted sets keep both an ordered structure and a lookup table, giving ordered ranges and fast score lookup.

for a middle

Name skiplist plus dict, map commands to each, and mention the listpack encoding below the thresholds.

for a senior

Add the spans/rank detail, the doubled per-element overhead, and the operational danger of unbounded ZRANGE on the single thread.

for a principal

Reason about whether a large sorted set is the right structure at all versus bucketing or an external index, given its per-element memory cost and reply-size risk.

## What a sorted set has to support A Redis sorted set holds unique members, each with a floating-point score, and its command set demands two very different access patterns: - **By member**: `ZSCORE key member`, `ZADD` updating an existing member, `ZREM`, `ZINCRBY`. These want a dictionary. - **By order**: `ZRANGE`, `ZRANGEBYSCORE`, `ZRANK`, `ZCOUNT`, `ZPOPMIN`, leaderboard slices. These want the data kept in score order with efficient rank arithmetic. No single classical structure gives both cheaply, so the `skiplist` encoding maintains two, kept in sync on every write. ## The skiplist A skiplist is a sorted linked list with extra forward pointers. Each node is assigned a random height; level 0 links every node in order, level 1 links roughly every second node, level 2 roughly every fourth, and so on. Searching starts at the top level and drops down whenever the next pointer would overshoot, so it skips exponentially decreasing distances — expected O(log n) for search, insert and delete, with no rebalancing logic. Ordering is by score, with the member string as tiebreaker, which is why members with equal scores come back in lexicographic order and why `ZRANGEBYLEX` is meaningful only when all scores are equal. Redis's variant adds a **span** to each forward pointer: the number of level-0 nodes it jumps over. Summing spans along a search path yields the element's rank without walking every node, giving O(log n) `ZRANK` and O(log n + m) range queries returning m elements. Why a skiplist rather than a balanced tree? It is simpler to implement correctly, its nodes are easy to traverse in both directions for range queries, the probabilistic balance needs no rotations, and the memory layout suits Redis's allocation patterns. The Redis author has said explicitly that simplicity and range-friendliness drove the choice; the asymptotics match a balanced tree. ## The dict Alongside it sits an ordinary hash table mapping member → score. Its job is O(1) answers to "is this member present and what is its score": - `ZSCORE` is a single dict lookup. - `ZADD` on an existing member must find the old score to remove the correct skiplist node before inserting the new one — without the dict, that lookup would be O(n). - `ZREM` and `ZINCRBY` likewise. The two structures share the member string object rather than duplicating the bytes, but the node and bucket overhead is genuinely paid twice. That makes large sorted sets the heaviest of the core types per element — a relevant fact when a leaderboard of tens of millions of entries is on the table. ## The compact alternative Below `zset-max-listpack-entries` (default 128) and `zset-max-listpack-value` (default 64 bytes for any member), Redis skips both structures and stores the sorted set as a **listpack**: one contiguous blob holding member, score, member, score, … in score order. Lookups and range queries are linear scans, but with at most 128 pairs in contiguous memory that is fast, and the memory saved is large — no skiplist nodes, no hash buckets, no per-element allocation headers. Crossing either threshold converts the value to the `skiplist` encoding, and as with all Redis encoding conversions it is **one-way**: shrinking the sorted set back below 128 members leaves it in the expensive encoding until the key is recreated. ## Practical consequences 1. **Memory planning.** If you hold many small sorted sets (per-user recent items, per-entity scores), keeping each under the listpack thresholds is worth real money. Design bucket sizes accordingly, and cap member length — one long member name flips the whole value. 2. **Big-O awareness on the single thread.** `ZADD`/`ZSCORE` are cheap at any size, but `ZRANGE key 0 -1` on a million-member sorted set returns a million elements and blocks the event loop for the duration. The skiplist makes the *seek* logarithmic; it does nothing about the size of the reply. Use bounded ranges, or `ZSCAN` for iteration. 3. **Diagnosis.** `OBJECT ENCODING` plus `ZCARD` plus `MEMORY USAGE` tells you immediately whether a sorted set's memory is dominated by structure overhead (skiplist, many small members) or by payload (few, long members).

  • If ZSCORE is O(1) thanks to the dict, why is ZRANGE key 0 -1 still dangerous on a large sorted set?
    Complexity of the seek is not the whole cost. ZRANGE with a full range is O(n) in elements returned: Redis must walk every node, serialise every member and score, and buffer the reply, all on the single command thread while every other client waits. A million-member leaderboard can block the server for a noticeable time and produce a huge reply buffer. Use bounded ranges such as ZRANGE key 0 99, or ZSCAN for incremental iteration.
  • Why did Redis choose a skiplist rather than a balanced binary search tree?
    They have the same O(log n) asymptotics, but the skiplist is much simpler to implement and to keep correct — probabilistic balancing needs no rotations or rebalancing cases. It also suits the workload better: level-0 is a plain doubly linked list, so range queries and reverse iteration are natural, and augmenting forward pointers with spans gives O(log n) rank queries with very little extra code.

The skiplist is the shelf where books stand in order, with express markers every few metres so you can jump; the dict is the catalogue card index telling you instantly which shelf position a given title occupies. A library needs both — browsing by order and finding one title by name are different jobs.

saying these in an interview costs you the question

  • Saying the skiplist encoding is a single structure
  • Claiming ZSCORE requires an O(log n) or O(n) search
  • Assuming a sorted set is memory-cheap at large sizes
  • Thinking the skiplist keeps members in insertion order rather than by (score, member)
  • Believing shrinking a large sorted set restores the listpack encoding

context