Apache Kafka — Interview Cheat Sheet
Producers, consumers, partitions, consumer groups, replication, and exactly-once semantics.
Interview tip Draw: Producer → Topic (partitions) → Consumer Group. Mention offset, replication factor, and why partitions enable parallelism.
① Core concepts
| Term | One-liner |
|---|---|
| Topic | Logical stream of records, split into partitions |
| Partition | Ordered, immutable append-only log — unit of parallelism |
| Offset | Monotonic position in partition — consumer tracks progress |
| Consumer group | Consumers cooperate — one consumer per partition max per group |
| Broker | Kafka server storing partition logs |
| Replication factor | Copies per partition — leader + followers (ISR) |
| ISR | In-sync replicas — caught up with leader |
| ZooKeeper / KRaft | Cluster metadata and controller election (KRaft replaces ZK) |
② Architecture diagram
Kafka cluster (simplified)
Producers
Brokers — Topic A (P0, P1, P2)
Consumer Group A
Consumer Group B
Data flow
Producer→Partition by key→Replicated log→Consumer poll→Commit offset
③ Producers — acks, keys, batching
| acks setting | Behavior | Durability |
|---|---|---|
| acks=0 | Fire and forget | May lose messages |
| acks=1 | Leader ack only | Lost if leader dies before replicate |
| acks=all (-1) | All ISR ack | Strongest — wait for min.insync.replicas |
Partition key: same key → same partition → ordering per key guaranteed.
Null key: round-robin across partitions — no ordering guarantee.
Batching: linger.ms + batch.size trade latency for throughput.
Null key: round-robin across partitions — no ordering guarantee.
Batching: linger.ms + batch.size trade latency for throughput.
④ Consumers & consumer groups
Consumer group: each partition assigned to at most one consumer in group — scale consumers up to partition count. More consumers than partitions → idle consumers.
Rebalance: triggered on consumer join/leave/crash — partitions reassigned (range, round-robin, sticky, cooperative sticky strategies).
Rebalance: triggered on consumer join/leave/crash — partitions reassigned (range, round-robin, sticky, cooperative sticky strategies).
| Commit mode | Pros | Cons |
|---|---|---|
| Auto commit | Simple | May commit before processing — at-most-once risk on crash |
| Manual sync commit | After successful process | Slower; still at-least-once without idempotency |
| Manual async commit | Non-blocking | Ordering of commits not guaranteed |
⑤ Ordering, retention & delivery semantics
| Guarantee | Scope | Requirement |
|---|---|---|
| Order | Within single partition only | Same key → same partition |
| At-most-once | Messages may be lost | Commit before process |
| At-least-once | Duplicates possible | Commit after process + idempotent consumer |
| Exactly-once | No dup, no loss (within Kafka txn scope) | Idempotent producer + transactions |
Retention: time-based (log.retention.hours) or size-based — Kafka is a log, not a traditional queue that deletes on read. Consumers track their own offsets.
⑥ Replication & fault tolerance
Leader partition serves reads/writes on a broker
Followers replicate from leader — join ISR when caught up
Leader failure → controller elects new leader from ISR
unclean.leader.election.enable=false — prefer availability vs data loss trade-off
min.insync.replicas + acks=all — prevent commit if too few replicas
Replication factor 3 in production — tolerate 2 broker failures with careful min.insync.replicas config.
⑦ Exactly-once semantics
EOS building blocks
Idempotent producer→Transactional API→Read-process-write
Idempotent producer: PID + sequence number dedupes retries on broker.
Transactions: atomic write to multiple partitions + consumer offset commit in same transaction — read-process-write pipelines.
Limit: EOS within Kafka ecosystem; external side effects still need idempotent sinks.
Transactions: atomic write to multiple partitions + consumer offset commit in same transaction — read-process-write pipelines.
Limit: EOS within Kafka ecosystem; external side effects still need idempotent sinks.
⑧ Kafka vs traditional queues
| Aspect | Kafka | RabbitMQ / SQS |
|---|---|---|
| Model | Distributed commit log | Queue — message deleted after ack |
| Replay | Yes — reset offset | No (unless DLQ/replay pattern) |
| Throughput | Very high sequential writes | Lower for very high volume |
| Ordering | Per partition | Single consumer per queue typically |
| Use case | Event streaming, analytics, CDC | Task queues, RPC-style messaging |
⑨ Scenario — order processing pipeline
Order events
Order Service (producer)
Topic: orders (key=order_id)
Payment consumer
Inventory consumer
Analytics sink
- Partition by order_id — all events for one order ordered
- Enough partitions for peak throughput (measure bytes/sec)
- Idempotent consumers — dedupe by event_id in store
- DLQ topic for poison messages after N retries
- Monitor consumer lag per partition
⑩ Tuning & interview Q&A
| Question | Answer sketch |
|---|---|
| Consumer lag high? | Scale consumers (≤ partitions), optimize processing, check rebalance storms, broker disk |
| Hot partition? | Skewed keys — salt keys or custom partitioner |
| How many partitions? | Target throughput / single partition throughput; plan ahead — hard to reduce |
| Why not exceed consumers vs partitions? | Extra consumers idle — wasted resources |
- Drew producer → topic/partitions → consumer group
- Explained acks=all and ISR
- Ordering per partition + key choice
- At-least-once vs exactly-once trade-offs
- Retention vs queue semantics