Fan-out on write and fan-out on read

Fan-out means turning one event into many downstream updates. In a feed, one new post must reach the timeline of every follower. You can do that work when the post is written (push), or when a viewer opens the feed (pull). Either way the work has to happen. Choosing a strategy only decides when it happens.

The big idea

Think of a newspaper. Push is delivering a copy to every doorstep each morning. The work is done up front, and reading is instant. Pull is a reader walking to the newsstand and collecting one copy from each paper they follow, every time they want to read. Writing is cheap, and reading is slow.

The analogy breaks when most doorstep copies go unread. In a feed, push writes are wasted on followers who never open the app.

Walking through it

These numbers come from the requirements page: about 700 posts per second on average, and about 200 followers per author.

Push, on average:

700 posts/s x 200 followers = 140,000 timeline writes/s

Push, for one celebrity:

1 post x 100,000,000 followers = 100,000,000 writes for one post
At 100,000 writes/s: 100,000,000 / 100,000 = 1,000 s, about 17 minutes

Pull, for one viewer:

viewer follows 200 accounts
each feed load reads 200 recent-post lists and merges them
at 200,000 loads/s: 200 x 200,000 = 40,000,000 key reads/s

Push makes a feed read one range query on the viewer's timeline. Pull makes every read touch hundreds of keys. Push is hard on writes, and pull is hard on reads.

Real systems combine the two. The next page, hybrid fan-out and celebrities, explains the mix.

Common mistakes

• Assuming push is free. Every follower gets a write, including accounts that never open the app.
• Pulling for everyone. At peak, pure pull turns feed loads into tens of millions of reads per second.
• Ignoring the celebrity tail. One post from a huge account can back up the queue for minutes.

Check yourself

1. An author with 50 followers posts once. How many timeline writes does push make?

   50, one for each follower's timeline.

2. Why is pull hard for a viewer with 200 followees at 200,000 loads per second?

   Each load reads about 200 author lists, so the system does about 40 million reads per second at peak.

Key takeaways

• Fan-out is the hard part of a feed. Storage is the easy part.
• Push makes reads cheap and writes expensive. Pull does the reverse.
• Celebrities break pure push, and high read volume breaks pure pull.
• The practical answer is a hybrid, which the next page covers.

Further learning

Next: hybrid fan-out and celebrities, the design most production feeds use.