Design News Feed
Facebook/Instagram feed — fan-out on write vs read, ranking, pagination, and the celebrity problem.
Interview tip Classic trade-off: push model for normal users, pull for celebrities with millions of followers. Mention hybrid approach and precomputed ranking features.
① Functional requirements
- Users create posts (text, image refs, metadata)
- Follow/unfollow other users (asymmetric graph)
- Home feed: ranked list of posts from followed users
- Infinite scroll pagination (cursor-based)
- Pull-to-refresh fetches new posts since last visit
- Like/comment counts displayed on feed cards (can be eventually consistent)
- Hide/mute users or posts
Out of scope (state in interview)
- Ads insertion and auction system
- Stories/ephemeral content (24h)
- Full comment thread rendering
- Content moderation ML pipeline
② Non-functional requirements
- Feed load < 2 seconds p99 on app open
- Support 300M DAU, 50M new posts/day
- Feed ranking refreshed within minutes of new engagement
- Highly available read path (99.9%)
- Eventual consistency acceptable for like counts
③ Back-of-the-envelope scale
Assumptions
- 300M DAU, 50M posts/day → ~580 posts/sec
- Avg 500 followers → fan-out on write: 580 × 500 = 290K writes/sec (heavy)
- Feed reads: 300M DAU × 5 opens/day = 1.5B reads/day → ~17K/sec
- Celebrity: 1 user × 50M followers = 50M fan-out writes per post (infeasible)
- Feed cache per user: 500 post IDs × 8B × 300M users = 1.2 TB (only active users cached)
- Post storage: 50M/day × 1KB × 365 × 3yr ≈ 55 TB
Hybrid fan-out is mandatory: push to follower feeds for users with < 10K followers; pull-merge at read time for celebrity posts. Precompute "celebrity list" (~1K users) dynamically by follower count.
④ High-level architecture
News Feed System
Mobile / Web Clients
API Gateway
Post Service
Social Graph Service
Feed Service
Fan-out Workers (Kafka)
Feed Cache (Redis)
Post DB (Cassandra)
Graph DB / adjacency lists
Ranking Service (ML features)
New post→Persist post→Fan-out worker→Update follower feeds
Feed cache stores sorted list of post IDs (not full posts). On read: fetch IDs from cache → batch get post details from post service. Ranking score computed at fan-out time or lazily on read.
⑤ Data flow & execution path
End-to-end execution flow
① Client→② API / LB→③ Core services→④ Cache + DB→⑤ Message queue→⑥ Async workers
Sync path: validate → authorize → read/write primary store
Async path: publish domain events → consumers (email, analytics, search index)
Read-heavy path: CDN / edge cache → regional cache → DB replica
Failure path: retry with backoff, DLQ, idempotent handlers
In interviews, trace one user action through this diagram. State what is synchronous (user waits) vs asynchronous (background), and where you enforce idempotency.
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| POST /v1/posts | Create post | {body, mediaIds[]} → postId; triggers async fan-out |
| GET /v1/feed?cursor=&limit=20 | Home feed | Returns ranked posts + next_cursor; merges celebrity pull |
| POST /v1/follow/{userId} | Follow user | Updates graph; may backfill recent posts to feed |
| DELETE /v1/follow/{userId} | Unfollow | Remove user posts from feed cache lazily |
| GET /v1/users/{id}/posts | Profile timeline | User's own posts; simpler than home feed |
⑦ Data model & storage
posts:
follows:
user_feed (Redis sorted set):
post_id, user_id, content, media_refs, created_at, like_countfollows:
follower_id, followee_id, created_at — adjacency list or graph DBuser_feed (Redis sorted set):
user_id → {post_id: rank_score} top 1000 entries| Store | What | Why |
|---|---|---|
| Cassandra | Posts by post_id and user_id | Durable post content; time-series per author |
| Redis Sorted Sets | Precomputed user feeds | ZADD on fan-out; ZREVRANGE on read; trim to 1000 |
| PostgreSQL / Neo4j | Social graph | Follow relationships; follower count for celebrity detection |
| Kafka | Fan-out job queue | Decouple post write from feed update; retry failed fan-outs |
⑧ Deep dive — core components
Hybrid fan-out (celebrity problem)
When user posts, check follower count. If < 10K: enqueue fan-out job — for each follower, ZADD post_id to their Redis feed sorted set. If ≥ 10K ("celebrity"): skip fan-out; store post only in author's timeline.
On feed read for user U: (1) ZREVRANGE U's precomputed feed. (2) Fetch recent posts from celebrities U follows (pull, max 50 celebs, last 24h). (3) Merge + re-rank by score. Cache merged result 60s.
Ranking pipeline
Score = w1×recency_decay + w2×engagement_rate + w3×affinity(user, author) + w4×content_type_boost. Precompute affinity from interaction history (likes, comments, DMs) nightly.
ML ranker (optional): gradient boosted trees on 100+ features; inference at read time < 50ms. A/B test ranking models. For interview: heuristic scoring at fan-out is sufficient.
Pagination & consistency
Cursor = (score, post_id) tuple for stable pagination under concurrent inserts. Not offset-based (breaks with new posts). "New posts" banner: compare client last_seen_ts with server max feed ts.
Staleness: fan-out async may lag 1–5s. Acceptable for social feed. Pull-to-refresh forces merge + bypass 60s read cache.
Unfollow and deleted posts
Unfollow: lazy removal — on next feed read, filter posts from unfollowed users; background job scrubs from Redis feed. Deleted post: tombstone in post DB; feed service filters tombstoned IDs on read; async purge from all feeds.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Fan-out | On write (push) | On read (pull) | Push for normal users; pull for celebrities |
| Feed storage | Redis sorted set | DB per-user table | Redis for speed; DB if Redis memory costly |
| Ranking | At fan-out time | At read time | Fan-out pre-rank for fast read; read-time for fresh engagement |
| Graph store | Adjacency lists in SQL | Dedicated graph DB | SQL to 1B edges; graph DB for complex queries |
| Consistency | Eventual feed update | Synchronous fan-out | Async fan-out standard; sync too slow |
⑩ 45-minute interview script
- 0–5 min: Clarify post types, follow model, ranking vs chronological
- 5–12 min: Scale math — posts/sec, fan-out writes, celebrity edge case
- 12–22 min: Draw architecture — post, graph, feed, fan-out workers
- 22–32 min: Deep dive hybrid fan-out; explain celebrity pull merge
- 32–38 min: Ranking signals, feed cache schema (sorted set)
- 38–42 min: Pagination cursors, pull-to-refresh
- 42–45 min: Unfollow, delete post propagation
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| 50M follower celebrity posts? | No fan-out; followers pull celebrity posts at read time; merge with precomputed feed |
| New user cold start? | Suggest popular accounts; content-based recommendations until graph builds |
| Feed shows stale like counts? | Acceptable; fetch fresh counts async; display cached with ~refresh icon |
| Mutual follow vs one-way? | This design is asymmetric (Twitter-style); symmetric requires different graph |
| How to backfill after follow? | Fetch followee's last N posts; ZADD to follower feed; async job |
| Sharding user feeds? | Shard Redis by user_id hash; each user feed entirely on one shard |
⑫ Revision checklist
- Fan-out on write vs read explained
- Celebrity hybrid approach
- Fan-out write math (posts × followers)
- Redis sorted set for feed cache
- Store post IDs not full posts in feed
- Ranking score formula mentioned
- Cursor-based pagination
- Kafka for async fan-out workers
- Feed load < 2s strategy (precompute)
- Unfollow lazy cleanup