Design API Gateway
Single entry point — auth, routing, rate limits, SSL termination, circuit breaking, and observability.
Interview tip Gateway vs service mesh: gateway at edge for north-south traffic; mesh for internal east-west. Mention Kong, AWS API Gateway, Envoy patterns. Keep gateway thin — no business logic.
① Functional requirements
- Route requests to correct backend service by path/host/header
- Authenticate requests (JWT, API key, OAuth2)
- Rate limit per consumer/API key
- SSL/TLS termination at gateway
- Request/response transformation (header injection, path rewrite)
- Circuit breaker: fail fast when backend unhealthy
- Centralized access logging and distributed tracing
- Canary routing: send 5% traffic to new service version
Out of scope (state in interview)
- Service mesh (Istio/Linkerd) for internal traffic
- Backend service implementation
- API versioning strategy (brief mention)
- GraphQL federation (mention as alternative)
② Non-functional requirements
- Gateway overhead < 10ms p99 added latency
- 99.99% gateway availability (no single point of failure)
- 100K RPS across gateway cluster
- Config changes deployed without downtime in < 60s
- Horizontally scalable — add nodes linearly
③ Back-of-the-envelope scale
Assumptions
- 100K RPS peak across all APIs
- 200 backend services; avg 500 RPS per service
- 50K API consumers with individual rate limits
- JWT validation: 100K crypto ops/sec → cache validated tokens 5 min
- Gateway cluster: 20 nodes × 5K RPS each = 100K capacity
- Config: 500 routes, 200 service definitions — 2MB total config
Gateway is stateless — scale horizontally behind LB. JWT public key cached locally; token validation cached in Redis (token_hash → claims, TTL = token expiry). Config pushed from control plane (etcd/Consul) with watch-based hot reload.
④ High-level architecture
API Gateway Architecture
External Clients
Mobile Apps
Cloud LB (TLS termination option)
API Gateway Cluster (Envoy/Kong)
Auth Service (JWT/OAuth)
Rate Limiter (Redis)
Service Registry (Consul/etcd)
Service A
Service B
Service C…
Logging (Kafka) + Tracing (Jaeger)
Request→Auth + rate limit→Route to service→Response + log
Request pipeline (middleware chain): TLS → access log → auth → rate limit → route → circuit breaker → proxy → response transform → metrics. Each plugin < 2ms. Config hot-reloaded via file watch or xDS (Envoy).
⑤ Data flow & execution path
End-to-end execution flow
① Client→② API / LB→③ Core services→④ Cache + DB→⑤ Message queue→⑥ Async workers
Sync path: validate → authorize → read/write primary store
Async path: publish domain events → consumers (email, analytics, search index)
Read-heavy path: CDN / edge cache → regional cache → DB replica
Failure path: retry with backoff, DLQ, idempotent handlers
In interviews, trace one user action through this diagram. State what is synchronous (user waits) vs asynchronous (background), and where you enforce idempotency.
⑥ API & interfaces
| Endpoint / flow | Purpose | Notes |
|---|---|---|
| /* (all client traffic) | Proxy to backend | Route table: path prefix → service name → upstream endpoints |
| POST /admin/routes | Add/update route | {path, service, methods, rate_limit, auth_required} |
| POST /admin/circuit-breaker/{service} | Configure CB | {failure_threshold, timeout_ms, half_open_requests} |
| GET /admin/health | Gateway health | Per-node status, config version, upstream health |
| GET /metrics (Prometheus) | Observability | request_count, latency_histogram, cb_state per service |
⑦ Data model & storage
routes:
services:
consumers:
id, path_pattern, service_id, methods[], auth_policy, rate_limit_id, canary_weightservices:
service_id, endpoints[], health_check_url, cb_configconsumers:
consumer_id, api_key_hash, rate_limit_tier, jwt_issuer| Store | What | Why |
|---|---|---|
| etcd / Consul | Route + service config | Watch-based push to all gateway nodes; versioned |
| Redis | Rate limit counters + JWT cache | Shared across gateway cluster |
| Kafka | Access logs | Async; every request logged with trace_id |
| Prometheus + Jaeger | Metrics + distributed traces | trace_id injected at gateway; propagated to backends |
⑧ Deep dive — core components
Authentication pipeline
JWT validation: extract Bearer token → check Redis cache (token_jti → claims) → on miss, verify signature with cached public key (fetched from auth service JWKS endpoint, refreshed hourly) → cache claims with TTL = token exp. API key: hash key → lookup consumer in local cache (30s TTL).
OAuth2: gateway redirects to auth service for token exchange; gateway never stores passwords. mTLS for service-to-service (optional layer).
Circuit breaker pattern
Per upstream service: CLOSED (normal) → OPEN (fail fast, return 503) after N failures in window → HALF_OPEN (allow 1 probe request) → CLOSED on success. Prevents cascade failure. Config: 5 failures in 10s → open for 30s.
Health checks: active probe every 10s + passive (track 5xx rate). Unhealthy endpoints removed from load balancer pool automatically.
Service discovery and routing
Gateway watches Consul/etcd for service endpoint changes. Route:
/api/users/* → user-service endpoints (round-robin). Canary: 95% to v1 endpoints, 5% to v2 (weighted random). Blue-green: flip weight 0→100 in one config update.Zero-downtime config deployment
Config versioned in etcd. Gateway nodes watch for changes → validate new config locally → atomic swap (double-buffering). Invalid config rejected — old config remains active. Rollback: revert etcd version; all nodes reload in <5s.
⑨ Trade-offs & alternatives
| Decision | Option A | Option B | Pick when |
|---|---|---|---|
| Gateway product | Envoy/Kong (self-hosted) | AWS API Gateway (managed) | Self-hosted for control; managed for ops simplicity |
| Auth | Gateway validates JWT | Backend validates | Gateway for defense-in-depth; backend still checks claims |
| TLS | At gateway | At LB + gateway | LB termination reduces gateway CPU |
| Config | etcd watch (push) | Poll every 30s | Push for fast updates; poll simpler |
| Logging | Sync log per request | Async Kafka | Async mandatory at 100K RPS |
⑩ 45-minute interview script
- 0–5 min: Clarify routing, auth, rate limit, observability needs
- 5–10 min: Scale — 100K RPS, 200 services, gateway cluster sizing
- 10–20 min: Architecture diagram — LB, gateway, registry, backends
- 20–28 min: Middleware pipeline walkthrough
- 28–35 min: JWT caching and circuit breaker deep dive
- 35–40 min: Service discovery and canary routing
- 40–45 min: Zero-downtime config deployment
⑪ Likely follow-up questions
| Question | Short answer |
|---|---|
| Backend down? | Circuit breaker opens; return 503 with Retry-After; alert ops; probe half-open after 30s |
| Gateway itself is bottleneck? | Add nodes horizontally; optimize hot plugins; move TLS to LB |
| How to handle 10MB request body? | Stream request to backend; don't buffer in gateway; set body size limit |
| Gateway vs service mesh? | Gateway: north-south client traffic; mesh: east-west service-to-service mTLS |
| DDoS at gateway? | WAF in front; global rate limit; IP blocklist; Cloudflare before gateway |
| API versioning at gateway? | Path prefix /v1/, /v2/ routes to different service versions; header-based alternative |
⑫ Revision checklist
- Single entry point for all client traffic
- Middleware pipeline defined
- JWT validation with caching
- Rate limiting per consumer
- Circuit breaker per upstream service
- Service discovery integration (Consul/etcd)
- Distributed tracing trace_id injection
- 100K RPS cluster sizing
- Config hot-reload without downtime
- Gateway vs service mesh distinction