Apache Spark — Interview Cheat Sheet
RDD vs DataFrame, lazy evaluation, shuffle, partitioning, caching, and job optimization.
Interview tip Shuffle is the enemy — explain wide transformations (groupBy, join) vs narrow (map, filter) and how to minimize data movement.
① Spark architecture overview
Spark cluster
Driver (SparkContext / SparkSession)
Cluster Manager (YARN / K8s / standalone)
Executor 1
Executor 2
Executor N
Driver: builds DAG, schedules tasks, tracks stages.
Executors: run tasks, store cached data, shuffle read/write.
Lazy evaluation: transformations build plan; actions (count, collect, write) trigger execution.
Executors: run tasks, store cached data, shuffle read/write.
Lazy evaluation: transformations build plan; actions (count, collect, write) trigger execution.
② RDD vs DataFrame vs Dataset
| API | Typing | Optimization | Status |
|---|---|---|---|
| RDD | Low-level, opaque objects | Manual — you optimize | Legacy — use when need fine control |
| DataFrame | Untyped rows + schema (Spark SQL) | Catalyst optimizer + Tungsten | Default for ETL |
| Dataset | Typed (Scala/Java) | Catalyst + encoder | Scala/Java — type-safe |
Python PySpark uses DataFrame API almost exclusively. Prefer DataFrame over RDD unless you need custom partition-level logic.
③ Transformations — narrow vs wide
| Type | Examples | Shuffle? |
|---|---|---|
| Narrow | map, filter, select, union (same partition count) | No — pipelined in same stage |
| Wide | groupByKey, reduceByKey, join, repartition, distinct | Yes — shuffle exchange |
Shuffle exchange
Map side write→Shuffle files→Reduce side read
groupByKey vs reduceByKey: reduceByKey combines locally before shuffle — almost always prefer reduceByKey over groupByKey + mapValues.
④ Partitioning & parallelism
| Operation | Effect |
|---|---|
| repartition(n) | Full shuffle — increase/decrease partitions evenly |
| coalesce(n) | Narrow — decrease partitions without full shuffle (if shuffle=false) |
| partitionBy(col) | Hash/range partition on write |
| spark.default.parallelism | Default partitions for shuffle operations |
Rule of thumb: 2–3× cores for partition count; too many tiny partitions → task overhead; too few → skew and memory pressure. Target 128MB–256MB per partition.
⑤ Join strategies
| Strategy | When | Risk |
|---|---|---|
| Broadcast hash join | Small table fits in memory (spark.sql.autoBroadcastJoinThreshold) | Driver/executor OOM if table too large |
| Sort-merge join | Large-large join | Expensive shuffle — both sides sorted |
| Shuffle hash join | Medium tables | Memory for hash table |
| Cartesian | Missing join condition bug | Explosion — always avoid accidentally |
Fix skew: salting hot keys, AQE skew join (Spark 3), isolate heavy key to separate processing.
⑥ Caching & persistence
| Level | Storage | Use |
|---|---|---|
| MEMORY_ONLY | Deserialized JVM objects | Fast if fits RAM |
| MEMORY_AND_DISK | Spill to disk on overflow | Default safe choice for iterative |
| DISK_ONLY | Disk | Too large for memory |
| MEMORY_ONLY_SER | Serialized bytes | Less memory, more CPU deserialize |
When to cache: DataFrame reused in multiple downstream branches (ML iterations, graph loops). Unpersist when done.
df.cache() = MEMORY_AND_DISK.⑦ Structured Streaming (brief)
Micro-batch model: treat stream as append table — trigger queries on new data. Checkpoint dir for fault tolerance and exactly-once sinks.
Watermark: handle late-arriving events in windowed aggregations.
Output modes: append, complete, update.
Watermark: handle late-arriving events in windowed aggregations.
Output modes: append, complete, update.
Streaming pipeline
Kafka source→Spark transform→Checkpoint→Delta / Parquet sink
⑧ Optimization checklist
- Filter early — push predicates before join
- Select only needed columns — avoid SELECT *
- Broadcast small dimension tables explicitly
- Replace UDFs with Spark SQL built-ins (Catalyst cannot optimize UDFs well)
- Avoid collect() on large data — use write or take/sample
- Enable AQE (Adaptive Query Execution) Spark 3+
- Fix data skew before scaling cluster
⑨ Debugging slow jobs
| Symptom | Likely cause | Fix |
|---|---|---|
| One task much slower | Data skew on key | Salt keys, AQE, isolate hot key |
| Executor OOM | Too much data per partition, large broadcast | Repartition, increase memory, reduce broadcast size |
| Many small tasks | Too many partitions | coalesce after filter |
| 4+ hour shuffle | Cartesian or missing filter | Check join condition, filter early |
| Spill to disk | Memory pressure | Increase executor memory or reduce partition size |
Use Spark UI: Stages tab → skewed task duration histogram; SQL tab → physical plan (broadcast hint visible).
⑩ Interview Q&A & revision
| Question | Answer sketch |
|---|---|
| What triggers shuffle? | Wide transformation requiring data redistribution across partitions |
| Lazy evaluation benefit? | Optimize full DAG — predicate pushdown, combine filters |
| RDD lineage? | Graph of transformations — rebuild lost partitions from lineage + checkpoints |
| Spark vs MapReduce? | In-memory iteratives, DAG scheduler, 10–100× faster for iterative ML/ETL |
| When Spark over pandas? | Data does not fit in memory, distributed cluster, production ETL pipelines |
- Explained narrow vs wide transformations
- Named shuffle cause and mitigation
- Compared RDD vs DataFrame
- Described broadcast join use case
- Listed skew debugging steps