Design Google News
Personalized news aggregation — crawl, classify, rank, and serve breaking stories at global scale.
Interview tip Separate ingestion (crawl + NLP) from serving (ranked feed). Mention edition by locale, dedup across publishers, and freshness vs quality ranking.
① Functional requirements
- Aggregate articles from publisher feeds and crawlers
- Personalized "For you" and topic feeds
- Full-text search across news corpus
- Breaking news detection and boost
- Multi-language and locale editions
- Publisher attribution and canonical URLs
② Non-functional requirements
- Feed p99 < 300ms
- Index new articles within minutes
- 99.9% availability
- Handle viral traffic spikes on breaking news
- Respect publisher crawl policies
③ Back-of-the-envelope scale
Assumptions
- 1M articles/day ingested
- 500M DAU
- 10K QPS feed reads
- Petabyte historical corpus
④ High-level architecture
Google News
Crawler + RSS ingest
NLP classify + dedup
Search index + ranker
Feed API + CDN
⑤ Data flow & execution path
Article pipeline
Crawl→Extract + NLP→Index→Rank feed
Dedup by URL hash
Embeddings for near-duplicate
User interest model offline
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| GET /feed | Personalized feed | cursor pagination |
| GET /search | News search | full-text + filters |
| POST /internal/index | Publisher push | authenticated |
⑦ Data model & storage
Domain-specific entities sharded by user_id or geographic key.
| Store | What | Why |
|---|---|---|
| Distributed store | Primary data | Sharded for scale |
| Kafka / Pub/Sub | Event log | Async pipelines |
| Object store | Media / blobs | GCS-style durability |
⑧ Deep dive — core components
Ranking
Blend recency, source authority, user clicks, diversity — logistic model trained offline, served in ms.
Dedup
Canonical URL + simhash on body text to cluster same story from multiple outlets.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Personalization | Deep model | Heuristic | Deep better CTR; heuristic faster to ship |
| Freshness | Stream index | Batch | Stream for breaking; batch cheaper |
⑩ 45-minute interview script
- 0–5 min: Requirements + Google-scale assumptions
- 5–12 min: Back-of-envelope QPS and storage
- 12–22 min: Architecture diagram
- 22–35 min: Deep dive on hot path
- 35–42 min: Failure modes and trade-offs
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| Fake news? | Source trust scores + human review queue + user feedback loop |
| Publisher paywall? | Show headline/snippet only; link out to publisher |
⑫ Revision checklist
- Crawl + ingest
- Dedup
- Ranker features
- Breaking boost
- Locale editions