Design ChatGPT System
LLM inference serving, conversation state, streaming tokens, GPU pool, and safety filters.
Interview tip Separate control plane (API, auth, billing) from data plane (GPU inference). Mention KV-cache, batching, streaming SSE, and rate limits per user.
① Functional requirements
- Multi-turn conversation with history
- Stream tokens to client (SSE)
- Model selection (fast/cheap vs capable)
- Stop/cancel generation
- Moderation on input and output
- Usage metering per user/org
② Non-functional requirements
- First token latency < 500ms p95
- 50K tokens/sec cluster throughput
- 99.9% API availability
- Isolate tenants — no cross-leak in KV cache
- Graceful degradation when GPU saturated
③ Back-of-the-envelope scale
Assumptions
- 1M active sessions — history stored, not all on GPU
- Context 8K–128K tokens — KV cache dominates VRAM
- 50K tok/sec ≈ hundreds of A100s with continuous batching
- Prompt cache for system prompts across users
④ High-level architecture
ChatGPT Serving
Clients
Chat API + auth
Session / history store
Router (model + GPU pool)
GPU inference workers
Moderation + billing
API loads recent history from DB, builds prompt, routes to inference pod with capacity. Continuous batching merges requests on same model.
⑤ Data flow & execution path
Chat completion stream
① Auth + rate limit→② Load history→③ Moderate input→④ GPU generate stream→⑤ Persist + bill tokens
KV cache reused for prefix (system prompt)
SSE chunk per token group
Cancel propagates to inference worker
Queue when GPUs full — return 429 or wait
Explain prefill vs decode phases. Prefill processes prompt in parallel; decode autoregressive — batch decode for efficiency.
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| POST /v1/chat/completions | Chat | stream=true SSE |
| POST /v1/chat/completions/cancel | Cancel | generation_id |
| GET /v1/models | List models | capability metadata |
| GET /v1/usage | Token usage | billing dashboard |
⑦ Data model & storage
Conversation: id, user_id, messages[]. Inference job: model, prompt_tokens, max_tokens, status. Usage: user_id, tokens, cost.
| Store | What | Why |
|---|---|---|
| PostgreSQL | Conversation history | encrypted at rest |
| Redis | Rate limits + session | token bucket |
| GPU VRAM | KV cache per request | ephemeral |
| Object store | Model weights | loaded per pod |
⑧ Deep dive — core components
Continuous batching and KV cache
Orca/vLLM style: batch new requests into running decode batch. Prefix KV cache shared when system prompts identical. VRAM limit caps concurrent contexts.
Safety and abuse
Input classifier before GPU — block jailbreak patterns. Output filter streaming — cut generation on policy hit. Per-user token rate limits.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Serving | Dedicated GPU | Serverless burst | Dedicated for steady chat load |
| History | Full context | Summarize old turns | Summarize saves tokens but may lose detail |
| Model | Single big model | Router small/large | Router saves cost on simple queries |
| Stream | Token stream | Full response | Stream better UX; harder output moderation |
⑩ 45-minute interview script
- 0–5 min: Chat + stream requirements
- 5–12 min: Token/sec GPU math
- 12–22 min: API + session store
- 22–32 min: Inference batching
- 32–40 min: Moderation + billing
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| RAG plugin architecture? | Tool retrieves docs → inject context chunk → model cites sources in reply |
| Fine-tuned per tenant? | Dedicated adapter weights loaded per request via tenant_id routing |
| Multi-region GPU pools? | Geo route to nearest pool; sticky session for long contexts; failover queue |
⑫ Revision checklist
- SSE streaming
- Conversation persistence
- GPU continuous batching
- KV cache / prefix cache
- Input/output moderation
- Rate limits per tier
- Cancel in-flight generation
- Token usage metering