skip to content

In a push-based social-network home timeline, how does a hybrid fanout model handle accounts with tens of millions of followers?

level: seniorimportance: must knowfreq 68%

answer

  1. skewed follower distribution
  2. one post, tens of millions of writes
  3. queue delay for everyone else
  4. outbox read at request time
  5. merge width stays small

basics

~20 s

Accounts above a follower threshold are excluded from fanout: their posts go only to their own recent-posts list. At read time the service merges the reader's pushed inbox with the recent posts of the few high-follower accounts they follow.

solid answer

~40 s

Pure push turns one post by an account with 50 million followers into 50 million inbox writes; at an illustrative 100,000 writes per second that single post takes about 500 seconds to finish, and every ordinary post queued behind it waits too. The **hybrid** marks accounts above a follower threshold as *pulled*: publishing writes the post once to that account's own recent-posts list and skips fanout. When a reader opens their timeline, the service reads their precomputed inbox, fetches the recent posts of the handful of pulled accounts they follow, merges everything by time, and hydrates the top page. Ordinary accounts stay on push, so the read cost stays bounded by the number of high-follower accounts a reader follows, which is usually small.

code

pseudocode · 6 lines
pseudocode
function homeTimeline(userId, limit):
  entries = inbox.range(userId, 0, limit)          // pushed post IDs, newest first
  for author in follows.pulledAccounts(userId):     // high-follower accounts only
    entries = entries + outbox.range(author, 0, limit)
  entries = sortByTimeDesc(dedupe(entries))
  return hydrate(entries.take(limit))                // IDs to full posts

go deeper

for a junior

Recall that very popular accounts are treated differently: their posts are not copied to followers but fetched when a follower reads.

for a middle

Walk through the read path: inbox read, outbox reads for pulled followees, time-ordered merge, dedupe, then hydration of the top page.

for a senior

Do the arithmetic out loud, 50 million writes at a stated throughput, and explain the shared-queue lag it causes before proposing the hybrid.

for a principal

Discuss the edges: readers following many pulled accounts, accounts crossing the threshold, and why the threshold is a cost policy rather than a constant.

## Why pure push breaks on huge audiences In **fanout-on-write**, each post is copied (as an ID) into the precomputed inbox of every follower. The cost of one post is therefore proportional to the author's follower count. For ordinary accounts that is fine: an author with 200 followers costs 200 writes. The distribution of follower counts is extremely skewed, though, and a small number of accounts have tens of millions of followers. Illustrative arithmetic, assuming a fanout fleet that can sustain 100,000 inbox writes per second in total: - One post by an account with 50,000,000 followers = **50,000,000 inbox writes**. - 50,000,000 / 100,000 per second = **500 seconds**, a little over 8 minutes, for that one post to reach the last follower. - If that account posts 10 times a day, it alone generates **500,000,000 writes per day**. The damage is not only the delay for that account's followers. While workers grind through its fanout, posts from ordinary accounts sit in the same queue, so **everyone's delivery lag rises**. A burst of posts from several such accounts can back the pipeline up for a long time. ## The hybrid model The fix is to stop pushing for the accounts that cause the problem: 1. Classify each account as **pushed** or **pulled**, typically by follower count against a threshold. 2. On publish by a pushed account: normal fanout into follower inboxes. 3. On publish by a pulled account: write the post ID once to that account's **recent-posts list** (its outbox) and skip fanout entirely. 4. On timeline read: combine the reader's inbox with the outboxes of the pulled accounts they follow. ```pseudocode function homeTimeline(userId, limit): entries = inbox.range(userId, 0, limit) for author in follows.pulledAccounts(userId): entries = entries + outbox.range(author, 0, limit) entries = sortByTimeDesc(dedupe(entries)) return hydrate(entries.take(limit)) ``` The read now does `1 + P` list reads, where `P` is the number of pulled accounts the reader follows. Because pulled accounts are by definition few, `P` is usually small for most readers, so the read path stays bounded. ## What each side pays | Account type | Write path | Read path contribution | |---|---|---| | Ordinary (pushed) | One write per follower, async | Already in the reader's inbox | | High-follower (pulled) | One write to its own outbox | One outbox read per reader, merged at request time | The outbox of a pulled account is read by many readers, which makes it a very popular read target; how to protect such a popular key is a caching concern of its own, separate from the fanout decision. ## Details that make the hybrid work - **Time-ordered IDs.** If post IDs sort by creation time, the merge can compare IDs directly instead of loading timestamps. - **Bounded merge width.** A reader who follows hundreds of pulled accounts reintroduces the pull problem. Designs cap the width, or push to that particular reader anyway. - **Deduplication.** An account that moved from pushed to pulled may have older entries in inboxes *and* in its outbox; the merge must dedupe by post ID. - **Crossing the threshold.** When an account becomes pulled, fanout stops; its earlier pushed entries remain in inboxes and age out through truncation. Hysteresis (different thresholds for switching each way) avoids flapping around the boundary. - **Freshness bonus.** Posts from pulled accounts appear instantly, since there is no fanout lag. ## Common misreadings - The hybrid does not mean "pull for everyone who follows a celebrity". Ordinary content still arrives by push; only the pulled accounts' posts are merged in. - It does not remove fanout lag for ordinary accounts; it removes the huge jobs that inflate that lag. - Follower count is the usual trigger, but posting rate and how many followers are active both change the real cost, so the threshold is a tuned policy rather than a universal number.

  • What happens to a hybrid timeline's read cost for a reader who follows hundreds of high-follower accounts?
    The read must fetch hundreds of outboxes and merge them, which is exactly the pull-path problem the hybrid was meant to avoid. Common mitigations are capping how many pulled outboxes a read merges, precomputing a merged list of pulled accounts for that reader, or treating that reader as a push target for those accounts.
  • Why must the hybrid merge deduplicate by post ID?
    An account that switched from pushed to pulled still has older posts sitting in followers' inboxes, and those same posts are also in its outbox. Without dedupe the reader sees duplicates. Deduplicating by post ID during the merge makes the transition safe without rewriting every inbox.
  • Why does one very large fanout job hurt followers of ordinary accounts too?
    Fanout workers share capacity. While they process tens of millions of inbox writes for one post, ordinary posts wait in the same queue or compete for the same workers, so the delivery lag for everyone rises. Removing the largest jobs from the push path keeps lag low for the whole system.

saying these in an interview costs you the question

  • Just add more fanout workers; huge accounts are only a throughput problem.
  • In the hybrid, every reader switches to pure fanout-on-read.
  • Pulled accounts' posts appear later than pushed posts.
  • The follower threshold is a fixed industry-standard number.
  • No deduplication is needed when an account changes mode.