Design Google Global Feature Flags
Rollout %, targeting rules, and consistent evaluation across millions of servers worldwide.
Interview tip Mention evaluation at client vs server, cache with TTL, sticky bucketing by user_id, and kill switch propagation in seconds.
① Functional requirements
- Define flags with on/off and percentage rollout
- Target by user_id, country, app version, cohort
- Real-time kill switch
- Audit who changed flag config
- SDK for servers and mobile clients
- Consistent assignment (same user always same variant)
② Non-functional requirements
- Evaluation < 1ms local cache
- Config propagation < 30s globally
- 99.99% availability — fail open or closed per flag
- Millions of flag reads per second
③ Back-of-the-envelope scale
Assumptions
- 10K flags
- 1B devices
- Consistent hash user → bucket for % rollout
④ High-level architecture
Feature Flags
Admin UI
Config service (CP)
CDN / edge cache
App SDK evaluate
⑤ Data flow & execution path
Flag evaluation
Load cached rules→Hash user_id→Match rules→Return variant
Push config deltas to edges
Sticky bucket: hash(user_id, flag_id)
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| GET /flags/{name} | Evaluate | SDK batches |
| PUT /admin/flags | Update rollout | audit log |
⑦ Data model & storage
Domain-specific entities sharded by user_id or geographic key.
| Store | What | Why |
|---|---|---|
| Distributed store | Primary data | Sharded for scale |
| Kafka / Pub/Sub | Event log | Async pipelines |
| Object store | Media / blobs | GCS-style durability |
⑧ Deep dive — core components
Sticky bucketing
hash(user_id, flag_name) % 100 < rollout_percent — user stays in cohort when % changes slightly.
Kill switch
Push empty allow list to all CDN edges; SDK polls every 30s or uses push channel.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Eval location | Client | Server | Client fewer RPCs; server more control |
| Fail mode | Fail closed | Fail open | Payments fail closed; UI experiments fail open |
⑩ 45-minute interview script
- 0–5 min: Requirements + Google-scale assumptions
- 5–12 min: Back-of-envelope QPS and storage
- 12–22 min: Architecture diagram
- 22–35 min: Deep dive on hot path
- 35–42 min: Failure modes and trade-offs
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| How roll out globally? | Regional cells + gradual feature rollout |
⑫ Revision checklist
- Sticky hashing
- Propagation speed
- Audit trail
- SDK cache TTL
- Kill switch tested