Design Distributed Logging
High-volume log ingestion, indexing, retention tiers, and search across petabytes.
Interview tip Separate hot write path (append-only) from query path. Mention Kafka, time-based indices, and retention (hot/warm/cold).
① Functional requirements
- Collect logs/metrics from services via agents or sidecars
- Parse structured and unstructured lines into fields
- Search by keyword, field filters, and time range
- Dashboards and saved queries
- Alerting on query thresholds (error rate spike)
- Per-team namespaces with ACLs
② Non-functional requirements
- Ingest 2M events/sec sustained
- Query p95 < 5s for 24h window on common filters
- 99.9% ingest availability
- 90-day retention with tiered storage cost
- No single tenant can starve shared pipeline
③ Back-of-the-envelope scale
Assumptions
- 2M events/sec × 500 bytes ≈ 1 GB/sec raw
- 7-day hot index ≈ 60TB before compression
- Time-based indices: hourly shards → easy drop/rollover
- 20 query nodes for parallel scan
- Compression 5× on cold tier → S3 Glacier
④ High-level architecture
Logging Platform
Log agents / Fluent Bit
Kafka (partition by service)
Stream parsers → enrich
Indexer cluster (hot)
Object store (warm/cold)
Query API + UI
Agents batch and compress. Kafka absorbs spikes. Indexers bulk-write time-sliced indices. ILM policy rolls hot → warm → delete.
⑤ Data flow & execution path
Log event lifecycle
① Agent batch→② Kafka partition→③ Parse/enrich→④ Bulk index→⑤ Query scan
At-least-once ingest with offset commits
Schema-on-read for unstructured logs
Hot tier: SSD indices last 48h
Cold tier: S3 + async restore for forensics
Trace one ERROR log line from host to searchable index. Mention duplicate handling via event_id.
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| POST /v1/logs | Agent ingest | gzip batch; 202 accepted |
| GET /search | Query DSL | time range + filters |
| POST /alerts | Create alert | runs query every N minutes |
| GET /indices | Retention status | per-tenant rollover |
⑦ Data model & storage
Event: timestamp, service, level, message, trace_id, host, custom fields. Index template per service family.
| Store | What | Why |
|---|---|---|
| Kafka | Ingest buffer | 24h retention for replay |
| Hot Elasticsearch | Recent logs | Fast search |
| S3 | Cold archive | Cheap long retention |
⑧ Deep dive — core components
Cardinality and field explosion
High-cardinality fields (user_id on every line) bloat indices. Use sampling for metrics, separate high-cardinality store, or indexed only on ERROR level.
Multi-tenant isolation
Namespace per team; RBAC on indices. Rate limit ingest per tenant. Noisy neighbor: dedicated Kafka topic quotas.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Ingest | Kafka | Direct HTTP | Kafka handles spikes and replay |
| Index | Elasticsearch | ClickHouse | ES for text; CH for analytics aggregates |
| Retention | Time indices | Single huge index | Time indices simplify delete |
| Parsing | At ingest | At query | Ingest parsing speeds search; query parsing flexible |
⑩ 45-minute interview script
- 0–5 min: Requirements and retention
- 5–12 min: Volume math
- 12–22 min: Pipeline diagram
- 22–32 min: Indexing and ILM tiers
- 32–40 min: Query performance and alerts
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| Trace_id correlation across services? | Shared trace_id field indexed; join queries or trace store (Jaeger) linked from log UI |
| PII scrubbing at ingest? | Regex + ML scrubber in parser stage before index; drop or hash sensitive fields |
| Live tail vs search? | Live tail reads Kafka consumer lag stream; search hits indexed store — different paths |
⑫ Revision checklist
- Agent batching + compression
- Kafka backpressure
- Time-based indices
- ILM hot/warm/cold
- Tenant ACLs
- Cardinality controls
- Alerting on scheduled queries
- Replay from Kafka on indexer failure