Design Google Photos Duplicate Detection
Find near-duplicate and burst photos — perceptual hashing, clustering, and user-facing suggestions.
Interview tip Pipeline: hash → bucket candidates → ML refine. Mention privacy (on-device vs cloud) and scale of billions of images.
① Functional requirements
- Detect visually similar photos and videos
- Group bursts into stacks
- Suggest "best" photo to keep
- On-upload processing
- User can dismiss suggestions
- Works across resolutions and crops
② Non-functional requirements
- Process upload within seconds
- Billions of photos corpus
- Low false positive on faces
- Privacy — user data isolated
③ Back-of-the-envelope scale
Assumptions
- 5B photos
- 50M uploads/day
- pHash 64-bit per image
④ High-level architecture
Photos Duplicate
Upload
Perceptual hash worker
Similarity index
Cluster + rank best
⑤ Data flow & execution path
Duplicate pipeline
Generate pHash→Hamming bucket→ML verify→Suggest stack
On-device hash optional
Embeddings for semantic similarity
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| POST /photos | Upload triggers pipeline | async |
| GET /stacks | Duplicate groups | user scoped |
⑦ Data model & storage
Domain-specific entities sharded by user_id or geographic key.
| Store | What | Why |
|---|---|---|
| Distributed store | Primary data | Sharded for scale |
| Kafka / Pub/Sub | Event log | Async pipelines |
| Object store | Media / blobs | GCS-style durability |
⑧ Deep dive — core components
Perceptual hash bucketing
Hamming distance ≤ 5 → candidate pair; CNN embedding for final merge.
Best photo selection
Score sharpness, faces smiling, exposure — lightweight on-device model.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Where compute | Cloud | On-device | On-device privacy; cloud more accurate |
| Index | LSH | Brute on recent | Recent window brute; archive LSH |
⑩ 45-minute interview script
- 0–5 min: Requirements + Google-scale assumptions
- 5–12 min: Back-of-envelope QPS and storage
- 12–22 min: Architecture diagram
- 22–35 min: Deep dive on hot path
- 35–42 min: Failure modes and trade-offs
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| How roll out globally? | Regional cells + gradual feature rollout |
⑫ Revision checklist
- pHash
- Hamming threshold
- ML second stage
- Privacy boundary
- User dismiss