Timeline storage

A timeline is the list of post IDs that a feed page reads from. Store one per user as a Redis sorted set, and the feed reads the newest IDs with a single range query. The timeline holds references, not full posts, and it can always be rebuilt from the source data. It is a cache of answers, not a record of truth.

The big idea

Posts live in the post store, which is the source of truth. The timeline is an index into that store, one per user, sorted by time. Like a book's index, it is small, quick to search, and can be rebuilt from the book itself.

Walking through it

Push post p_987 into user 42's timeline, then read the newest 20 IDs:

ZADD timeline:U42 1697300000 p_987   # score = timestamp, member = post ID
ZREVRANGE timeline:U42 0 19          # newest 20 post IDs, newest first

Hydrating a page takes three steps:

1. ZREVRANGE returns the post IDs.
2. MGET fetches the post bodies from the post cache.
3. MGET fetches like counts from the counters service, which is eventually correct.

Never join the social graph live on every card. Keep the graph out of the read path.

Here is a capacity check. If each feed load does one timeline read, three celebrity reads and two MGET calls, that is six Redis operations per load. At 200,000 loads per second, that's about 1.2 million operations per second. That is comfortable across a sharded cluster and far too much for one box. So timeline keys are partitioned by userId, the post cache by postId, and the few celebrity keys are replicated because they are the hottest.

Operating rules:

• Cap the length at roughly 800 to 2,000 entries. Older posts fall back to an on-demand pull or a colder store.
• Backfill on follow. When someone follows an account, copy its recent posts into the new follower's timeline, asynchronously.
• Filter on unfollow. Remove the posts lazily on read, or clean them up asynchronously.
• Rebuild on a miss. Keep timelines for recently active users. For everyone else, rebuild from the follow graph and recent posts when they return.

Common mistakes

• Storing full post bodies. A post is copied into many timelines. Store IDs and hydrate on read.
• Treating the timeline as the source of truth. Losing the Redis cluster should cost a rebuild, not data. Keep the post store durable.
• Leaving timelines uncapped. Old entries pile up for accounts nobody opens.

Check yourself

1. Why store post IDs rather than whole posts in the timeline?

   A post is copied into many timelines, so storing bodies multiplies storage. IDs are small, and bodies are hydrated from the post cache on read.

2. What happens if the Redis timeline cluster is lost?

   Timelines are derived, so they are rebuilt from the follow graph and posts. That is slow for hot users if nothing is prepared, but no content is lost.

Key takeaways

• A timeline is a per-user sorted set of post IDs, cheap to read with one range query.
• Hydrate post bodies and counts separately, from caches.
• Cap timelines, backfill on follow, and rebuild on a miss.
• Treat timelines as rebuildable, and invest in backfill and lag monitoring.

Further learning

Next: hybrid fan-out and celebrities, which explains how timelines get filled for ordinary authors and celebrities.