Design Twitter
Home timeline, tweets, retweets, search, and trending — hybrid fan-out with Snowflake IDs and real-time search.
Interview tip Same fan-out hybrid as news feed. Add tweet ID as Snowflake, separate search index for @mentions and hashtags, and retweet deduplication in timeline.
① Functional requirements
- Post tweet (text, optional media, reply_to tweet_id)
- Retweet and quote-tweet
- Home timeline: tweets from followed users, ranked by recency + engagement
- User profile timeline (all tweets by user)
- Search tweets by keyword, hashtag, @mention
- Trending topics (top hashtags by velocity in last hour)
- Like, reply counts on tweet cards
Out of scope (state in interview)
- Direct messages (separate system)
- Ads and promoted tweets
- Spaces / live audio
- Algorithmic "For You" feed ML (mention only)
② Non-functional requirements
- Timeline load < 2s p99
- Tweet post acknowledged < 500ms
- Search results < 500ms for recent tweets
- 500M tweets/day write throughput
- Durable — tweets never lost after ack
③ Back-of-the-envelope scale
Assumptions
- 500M tweets/day → ~5.8K tweets/sec avg, ~30K/sec peak
- Avg 200 followers → 5.8K × 200 = 1.16M timeline writes/sec (push fan-out)
- Timeline reads: 400M MAU × 10 sessions/day × 3 timeline loads = 12B/day → ~140K/sec
- Search index: 500M new docs/day; index size ~500GB/year compressed
- Trending: aggregate 50M unique hashtags/hour from stream
- Snowflake IDs: 64-bit, time-sortable, 4096 IDs/ms per machine
Users with > 1M followers skip push fan-out (pull at read). Timeline cache stores tweet IDs in Redis; hydrate tweet bodies from tweet store on read. Retweets stored as lightweight reference, not full copy.
④ High-level architecture
Twitter Architecture
Clients
API Layer
Tweet Service
Timeline Service
Search Service (Elasticsearch)
Graph Service
Fan-out Workers
Trending Aggregator (Flink)
Tweet Store (MySQL/Cassandra)
Timeline Cache (Redis)
Media CDN
Post tweet→Snowflake ID→Store + index→Fan-out timelines
Tweet service owns tweet CRUD. Timeline service owns fan-out and read merge. Search is async index via Kafka → Elasticsearch. Graph service stores follow edges and follower counts for celebrity detection.
⑤ Data flow & execution path
End-to-end execution flow
① Client→② API / LB→③ Core services→④ Cache + DB→⑤ Message queue→⑥ Async workers
Sync path: validate → authorize → read/write primary store
Async path: publish domain events → consumers (email, analytics, search index)
Read-heavy path: CDN / edge cache → regional cache → DB replica
Failure path: retry with backoff, DLQ, idempotent handlers
In interviews, trace one user action through this diagram. State what is synchronous (user waits) vs asynchronous (background), and where you enforce idempotency.
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| POST /v1/tweets | Create tweet | {text, mediaIds?, replyTo?, quoteOf?} → tweetId |
| POST /v1/tweets/{id}/retweet | Retweet | Creates retweet reference; fan-out retweet ID to followers |
| GET /v1/timeline/home?cursor= | Home timeline | Merged push+pull; cursor = last tweetId |
| GET /v1/users/{id}/tweets | Profile timeline | Chronological tweets by user |
| GET /v1/search?q=&type= | Search | Full-text + hashtag + mention filters |
| GET /v1/trends | Trending | Top 20 hashtags by velocity; cached 60s |
⑦ Data model & storage
tweets:
timeline (Redis list per user): ordered tweet_ids, max 800 entries
follows:
tweet_id (Snowflake PK), user_id, text, type (original|retweet|reply|quote), ref_tweet_id, created_attimeline (Redis list per user): ordered tweet_ids, max 800 entries
follows:
follower_id, followee_id, followee_follower_count (denormalized)| Store | What | Why |
|---|---|---|
| MySQL / Cassandra | Tweet bodies | Snowflake PK; secondary index by user_id + time |
| Redis Lists / Sorted Sets | Home timelines | LPUSH on fan-out; LRU trim to 800 tweets |
| Elasticsearch | Search index | Inverted index on text, hashtags, mentions; near-real-time |
| Kafka + Flink | Trending pipeline | Windowed count of hashtags; top-K every minute |
⑧ Deep dive — core components
Snowflake tweet IDs
64-bit ID: 41 bits timestamp (ms) + 10 bits machine ID + 12 bits sequence. Time-sortable — timeline pagination by tweet_id works as cursor. 4096 tweets/ms per machine; scale machines horizontally. Clock rollback handling: wait or use spare bits.
Hybrid timeline fan-out
On tweet: if author has < 1M followers, fan-out worker LPUSH tweet_id to each follower's Redis timeline. If ≥ 1M, skip fan-out — mark author as "celebrity".
On home timeline read: (1) LRANGE follower Redis timeline (800 ids). (2) For each celebrity followed, fetch last 20 tweets from celebrity timeline. (3) Merge by tweet_id desc, dedupe retweets, return page.
Search & trending
Tweet creation event → Kafka → Elasticsearch indexer. Index fields: text (analyzed), hashtags (keyword), mentions (keyword), user_id, created_at. Search within 5s of tweet post (near-real-time ES refresh).
Trending: Flink tumbling 5-min window counts hashtag occurrences; score = count × velocity_boost; top 20 stored in Redis; global + per-location trends.
Retweet handling
Retweet stores only
ref_tweet_id + retweeter user_id — no text duplication. Fan-out pushes retweet's Snowflake ID to followers. On display, hydrate original tweet. Dedupe: if follower already has original in timeline, optionally skip retweet (product decision).⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Timeline | Push fan-out | Pull on read | Push for normal; pull celebrities only |
| Tweet store | MySQL sharded | Cassandra | MySQL to ~50K writes/sec/shard; Cassandra beyond |
| Search | Elasticsearch | Custom inverted index | ES standard; custom only at extreme scale |
| Retweet storage | Reference only | Full copy | Reference saves storage; copy faster read |
| Trending | Stream (Flink) | Batch hourly | Stream for real-time trends; batch cheaper |
⑩ 45-minute interview script
- 0–5 min: Clarify tweet types, timeline vs search, trending
- 5–12 min: Scale — 500M/day, fan-out math, celebrity threshold
- 12–22 min: Architecture — tweet, timeline, search, graph services
- 22–30 min: Snowflake IDs and hybrid fan-out deep dive
- 30–36 min: Search indexing path; trending pipeline
- 36–42 min: Retweet model, timeline pagination
- 42–45 min: Viral tweet hot key mitigation
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| Delete tweet propagation? | Tombstone in tweet store; async purge from timelines; search index delete |
| @mention notification? | Parse mentions on write; async notify mentioned users via notification service |
| Timeline for user following 5000 accounts? | Cap pull celebrities; sample; or rank follows by engagement |
| Elasticsearch falls behind? | Degrade search to recent-only from tweet DB; alert ops; scale indexers |
| Quote tweet in timeline? | Fan-out quote tweet ID; display embeds original with quoted context |
| Rate limit tweets? | Per user 100/day API limit; 2400/day for verified; token bucket in tweet service |
⑫ Revision checklist
- Snowflake ID for tweets explained
- Hybrid push/pull fan-out
- Celebrity threshold (~1M followers)
- Separate search index (Elasticsearch)
- Trending via stream aggregation
- Retweet as reference not copy
- Timeline stores IDs not bodies
- 500M tweets/day scale math
- Graph service for follow edges
- Kafka for async indexing + fan-out