System Design

Design Notification System

Push, email, SMS — templates, user preferences, priority queues, retries, and delivery guarantees.

Interview tip Multi-channel fan-out from one event. User prefs table gates each channel. Retry with exponential backoff + DLQ. Idempotent notification IDs prevent duplicate sends.

① Functional requirements

Send notification via push, email, or SMS from single API call
User preferences: enable/disable per channel; quiet hours (no push 10pm–8am)
Template engine: parameterized templates with i18n/locale support
Priority: transactional (OTP, receipt) vs marketing (promo)
Schedule future notifications (reminder in 24h)
Delivery status tracking: sent, delivered, failed, bounced
Batch notifications (digest email: "5 friends liked your photo")

Out of scope (state in interview)

In-app notification inbox UI
A/B testing notification content
Rich push (images, actions) — mention only
Notification analytics dashboard (brief)

② Non-functional requirements

Transactional notifications delivered < 30s p99
1B notifications/day → ~12K/sec avg, ~100K/sec peak
At-least-once delivery with idempotency (no duplicate user-visible sends)
99.9% availability on ingestion API
Graceful degradation if one channel provider is down

③ Back-of-the-envelope scale

Assumptions

1B notifications/day → ~12K/sec avg; peak 100K/sec (flash sale, viral event)
500M users; avg 2 devices/user → 1B push tokens
Channel split: 60% push, 30% email, 10% SMS
Email: 3.6K/sec → SendGrid limit ~10K/sec (OK with batching)
SMS: 1.2K/sec avg; Twilio ~100/sec/account → need 12 accounts or queue smoothing
Template storage: 10K templates × 50 locales = 500K rendered variants (cached)

Priority queues: transactional traffic on dedicated high-priority Kafka topic processed first. Marketing batched into digests (reduce 10 events → 1 email). SMS always queued with rate limiter matching provider capacity.

④ High-level architecture

Notification System

Backend Services (events)

Notification API (ingest)

Scheduler (delayed notifications)

Notification Service (route + template)

User Preferences DB

Push Queue

Email Queue

SMS Queue (rate limited)

Push Workers → FCM/APNs

Email Workers → SendGrid

SMS Workers → Twilio

Delivery Status DB + DLQ

Event→Check prefs→Render template→Enqueue channel

Ingestion API is sync (ack immediately after Kafka write). All delivery async. notification_id (UUID) dedupes — check delivery_status DB before send. Quiet hours: scheduler holds push until window opens.

⑤ Data flow & execution path

End-to-end execution flow

① Client→② API / LB→③ Core services→④ Cache + DB→⑤ Message queue→⑥ Async workers

Sync path: validate → authorize → read/write primary store
Async path: publish domain events → consumers (email, analytics, search index)
Read-heavy path: CDN / edge cache → regional cache → DB replica
Failure path: retry with backoff, DLQ, idempotent handlers

In interviews, trace one user action through this diagram. State what is synchronous (user waits) vs asynchronous (background), and where you enforce idempotency.

⑥ API & interfaces

Endpoint / flow	Purpose	Notes
POST /v1/notifications/send	Send notification	{userId, templateId, data{}, channels[], priority, idempotencyKey}
POST /v1/notifications/schedule	Schedule future	{sendAt, ...sendPayload}
GET /v1/users/{id}/preferences	Get channel prefs	Returns {push, email, sms, quietHours}
PUT /v1/users/{id}/preferences	Update prefs	Opt-in/out per channel
GET /v1/notifications/{id}/status	Delivery status	Per channel: queued\|sent\|delivered\|failed
POST /v1/templates	Create template	{name, channels: {push, email, sms}, locales{}}

⑦ Data model & storage

notifications: notification_id, user_id, template_id, data JSON, priority, status, created_at

delivery_attempts: notification_id, channel, attempt, status, provider_id, error

user_preferences: user_id, push_enabled, email_enabled, sms_enabled, quiet_hours, locale

Store	What	Why
Kafka (per-priority topics)	Notification queue	Transactional topic prioritized; marketing batched
PostgreSQL	Delivery status + idempotency	notification_id unique; track per-channel attempts
Redis	User prefs cache + SMS rate limiter	Prefs cached 5 min; SMS token bucket per provider
S3	Rendered template cache	Pre-rendered locale variants; invalidate on template update

⑧ Deep dive — core components

Idempotency and deduplication

Client sends idempotencyKey (or server generates notification_id). On ingest: INSERT INTO notifications ON CONFLICT DO NOTHING. Workers check delivery_attempts before sending — if status=sent, skip. Prevents duplicate on Kafka replay or worker retry.

Template rendering and i18n

Templates stored as Handlebars/Jinja with variables: "{{userName}} liked your photo". On send: fetch user locale → load template variant → render with data → cache rendered output keyed by (template_id, locale, data_hash) for 5 min.

Batch digest: accumulator groups N similar events per user per hour → single "5 people liked your photo" email instead of 5 separate emails.

Retry, backoff, and DLQ

On delivery failure (provider 5xx, timeout): retry 3× with exponential backoff (1s, 10s, 60s). After max retries → Dead Letter Queue (DLQ). Ops dashboard alerts on DLQ depth. Manual replay tool for DLQ after provider recovery.

Provider-specific: APNs invalid token → mark device token inactive (don't retry). Email bounce → suppress email channel for user.

Quiet hours and scheduling

Push during quiet hours (10pm–8am user local time): scheduler writes to delayed queue with execute_at. Cron worker scans due notifications every minute. Transactional (OTP) bypasses quiet hours. Timezone stored per user.

⑨ Trade-offs & alternatives

Decision	Option A	Option B	Pick when
Delivery	At-least-once + idempotent	Exactly-once	At-least-once practical; exactly-once needs distributed tx
Marketing	Batch digest	Real-time per event	Digest reduces fatigue and email cost
SMS rate limit	Queue + smooth	Multiple provider accounts	Queue simpler; multi-account for burst
Template	Pre-rendered cache	Render on send	Cache for repeated templates; render for dynamic
Priority	Separate Kafka topics	Single queue with priority field	Separate topics guarantee transactional SLA

⑩ 45-minute interview script

0–5 min: Clarify channels, preferences, priorities, templates
5–12 min: Scale — 1B/day, channel split, provider limits
12–22 min: Architecture — ingest, service, per-channel queues, workers
22–30 min: Idempotency and dedup deep dive
30–36 min: Retry/DLQ and provider failure handling
36–42 min: Quiet hours scheduler; batch digest
42–45 min: SMS rate limit smoothing

⑪ Likely follow-up questions

Question	Short answer
User disables push?	Check preferences before enqueue; skip push channel; still send email if enabled
10K SMS/sec spike?	Queue all; rate limiter drains at 100/sec; delay acceptable for marketing; alert ops
Push token invalid?	APNs/FCM returns 410; mark token dead; stop future push to that device
Duplicate notification on retry?	idempotencyKey + delivery_attempts table prevents re-send
Global quiet hours vs per-user timezone?	Per-user timezone stored; scheduler converts to UTC execute_at
How to test without sending real SMS?	Sandbox provider mode; override recipient in staging; mock workers

⑫ Revision checklist

Multi-channel from single event API
User preferences gate each channel
Idempotency key prevents duplicates
Separate queues per channel
Priority: transactional vs marketing
Retry with exponential backoff + DLQ
Template engine with i18n
Quiet hours scheduler
Batch digest for marketing
SMS rate limit smoothing

notificationsqueuespushemailreliability