SYSTEM DESIGN:Lesson 20: Scaling Feeds Past the Celebrity Problem

50 million people, one Kohli tweet

Lesson 19 gave you push vs pull. Now scale it. IPL Super Over ends. Virat Kohli posts a two-word tweet. Sixty million accounts follow him. Your fan-out worker starts writing to sixty million Redis lists. The queue depth graph looks like a cliff. Meanwhile, forty-nine million normal users are opening their feeds — each expecting results in under 200 ms.

This chapter: hybrid fan-out, Redis feed cache, ranking pipeline, napkin math for 50M DAU, and the pushback interviewers throw when you say "just cache it."

Celebrity tweet fan-out to millions of followers

Two words from Kohli. Sixty million Redis writes. Good luck.

Fan-out on write vs fan-out on read

Push for your college friend. Pull for Kohli. Hybrid is not cheating — it is the product.

Napkin math — 50M DAU

State these numbers before drawing boxes.

Assumptions:

• 50M daily active users
• Each user opens feed 30× per day (commute, lunch, IPL breaks, bedtime)
• Each user posts 2 tweets per day on average (power law: most post 0, some post 50)
• Average following count: 200 accounts
• Celebrity threshold: 10,000 followers → switch to pull

Reads (feed opens):

50M users × 30 feed opens / 86,400 sec ≈ 17,400 reads/sec average
Peak (3× evening + IPL): ~52,000 reads/sec

Writes (new tweets):

50M × 2 posts / 86,400 sec ≈ 1,160 writes/sec average
Peak (3×): ~3,500 writes/sec

Read-to-write ratio: ~15:1 on feed opens alone. Add profile views, likes — easily 30:1 or higher. Optimize the read path first.

Fan-out writes (push path only):

Average user: 200 followers → 200 cache writes per tweet
1,160 tweets/sec × 200 ≈ 232,000 Redis writes/sec (distributed across workers)
Kohli tweet: 60M writes → async queue, minutes to complete — NOT synchronous

The celebrity tweet is an outlier event, not steady-state QPS. Design for steady-state; isolate outliers with hybrid fan-out.

Hybrid fan-out architecture

On post by user U:
  if U.follower_count < CELEBRITY_THRESHOLD:
    async push tweet_id to each follower's Redis feed (ZSET)
  else:
    insert tweet only — followers merge at read time

On feed read by user V:
  1. Fetch pre-built feed from Redis (pushed tweets)
  2. Fetch recent tweets from celebrity accounts V follows (pull)
  3. Merge, rank, return top N

Threshold tuning: 10k is a common interview number. Production teams A/B test it. Too low → too many pull merges on read. Too high → fan-out workers drown.

CELEBRITY_THRESHOLD = 10_000

def on_new_tweet(tweet: Tweet):
    db.insert(tweet)
    follower_count = db.count_followers(tweet.author_id)
    if follower_count < CELEBRITY_THRESHOLD:
        queue.publish("fanout", { "tweet_id": tweet.id, "author_id": tweet.author_id })
    # celebrities: no fan-out job

def get_feed(user_id: int, limit: int) -> list[Tweet]:
  pushed = redis.zrevrange(f"feed:{user_id}", 0, limit * 2)
  celeb_ids = db.get_celebrity_following(user_id)  # follows with >10k followers
  pulled = db.get_recent_tweets(celeb_ids, limit=limit)
  merged = merge_and_rank(pushed, pulled)
  return hydrate(merged[:limit])

Redis feed cache

Pre-built feeds live in Redis sorted sets — score = timestamp or tweet_id for ordering.

Key:   feed:{user_id}
Type:  ZSET
Score: tweet created_at (unix ms)
Member: tweet_id

Fan-out worker:

def fanout_worker(tweet_id: int, author_id: int):
    followers = db.get_followers(author_id)  # paginate in chunks of 1000
    score = tweet.created_at.timestamp()
    pipe = redis.pipeline()
    for fid in followers:
        key = f"feed:{fid}"
        pipe.zadd(key, {tweet_id: score})
        pipe.zremrangebyrank(key, 0, -501)  # cap at 500 tweets
    pipe.execute()

Cap feed depth at 500–1000 tweets. Older posts fall off — user scrolls to "load more" which hits DB for archive.

Redis cluster shards by user_id — each user's feed key lands on one node. This is a natural key-value workload (Lesson 11, Lesson 12).

Ranking and reranking pipeline

Chronological feed is the MVP. Real X/Instagram feeds are ranked — engagement prediction, recency decay, "you might have missed this."

Two-stage pipeline (common pattern):

Stage 1 — Candidate generation (cheap, broad):
  - Pushed tweets from Redis
  - Pulled celebrity tweets
  - ~500 candidates

Stage 2 — Ranking / reranking (expensive, narrow):
  - ML model scores each candidate: P(like), P(reply), P(dwell_time)
  - Boost friends, demote seen posts, inject ads slot 4
  - Return top 20

Interview script: "I start chronological. At scale, add offline feature store + lightweight model for ranking. Stage 1 is system design; Stage 2 is ML infra — I mention the interface, not train the model."

Reranking signals (name a few):

• Recency — exponential decay, half-life ~6 hours
• Engagement — likes/retweets in first hour
• Relationship — close friend vs acquaintance
• Already seen — dedupe from impression log

Do not claim you will build PageRank in 45 minutes. Say "engagement-weighted score with recency bias" and move on.

Hot users and cache warming

Hot user = account whose tweets fan out to huge follower counts OR account whose feed is read constantly (trending page, official IPL handle).

Problems:

• Hot author → fan-out queue lag during viral moment
• Hot reader → Redis feed key hit 100×/sec if they refresh obsessively during match

Mitigations:

1. Cache warming — pre-populate feed for users who open app daily at 8 AM
2. CDN-style edge cache for public profiles (not home feed — too personalized)
3. Rate limit feed refresh API — 1 req/sec per user is plenty
4. Separate "viral fan-out" pool with more workers, auto-scale on queue depth

Cache warming sounds fancy. Implementation: cron job at 7:55 AM IST calls get_feed for top 10M DAU users, populates Redis before they wake up. Wasteful for inactive users — segment by last_active_at.

Media tweets at scale

Text tweet: store row in Postgres, push tweet_id to feeds.

Image/GIF tweet:

1. Client uploads to S3 via presigned URL
2. Post service stores tweet with media_url reference
3. Fan-out pushes tweet_id only — not the image bytes
4. Feed API returns media_url; client fetches from CDN

Never fan-out binary blobs. Fan-out pointers. CDN handles IPL meme traffic, not your Redis cluster.

Interview pushback — prepare answers

"What if the feed is stale?"

Async fan-out means 1–5 second delay for normal users. Say it proactively:

• Acceptable for social timeline — not a stock ticker
• Show tweet immediately on author's own profile (read from tweets table)
• WebSocket or polling for "new tweets available" banner — user taps to refresh

"How do you rank the feed?"

• MVP: chronological merge by timestamp
• V2: weighted score = recency + engagement + social graph proximity
• Heavy ML offline; online serving uses precomputed features from feature store

"What about media / video?"

• Upload to object storage, serve via CDN
• Feed stores metadata only; lazy-load media on scroll
• Video transcoding async — tweet goes live with poster image, video ready in 30 sec

"Celebrity unfollowed — stale cache?"

def unfollow(follower_id, followee_id):
    db.delete_follow(follower_id, followee_id)
    # Do NOT scan Redis removing old tweets — too expensive
    # Filter at read time: exclude authors not in following set

Lazy invalidation beats eager cleanup for social graphs. Consistency within minutes is fine.

Architecture diagram (verbal)

                    ┌─────────────┐
  POST /tweets ────►│ Post Service│──► Postgres (tweets)
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐     ┌──────────────┐
                    │ Kafka/SQS   │────►│ Fan-out      │──► Redis feed:{uid}
                    └─────────────┘     │ Workers      │
                                          └──────────────┘

  GET /feed ───────► Feed Service ──► Redis (pushed) + Postgres (celebrity pull)
                           │
                    ┌──────▼──────┐
                    │ Ranker      │──► top 20 tweets
                    └─────────────┘

Monitoring at scale

• Feed read p99 — > 300 ms means Redis miss or slow celebrity merge
• Fan-out queue depth — growing = workers underprovisioned or celebrity slip-through
• Redis memory — feed ZSETs × 500 members × 50M users = plan capacity
• Stale feed complaints — track fan-out lag p95 (time from post to cache)

The 5-minute interview answer

1. Clarify: post + home feed, 50M DAU, read-heavy
2. Hybrid fan-out: push if < 10k followers, pull celebrities at read
3. Redis ZSET per user for pushed tweets; cap at 500
4. Async fan-out via queue — POST returns before cache writes finish
5. Feed read: merge Redis + celebrity pull → rank → return 20
6. Napkin math: ~52k feed reads/sec peak, ~3.5k writes/sec peak
7. Media via S3 + CDN; ranking V2 with ML feature store

KV store connection

Feed cache is a distributed key-value problem: key = user_id, value = ordered list of tweet_ids. Sharding, replication, and hot-key handling from Lesson 11 and Lesson 12 apply directly.

What comes next

Feeds are broadcast. Chat is point-to-point (or small group) with delivery guarantees and online presence. Different beast.

Continue here: Chat System Core — WhatsApp Group on Diwali Night.