Here is a detailed step-by-step lineage of how a Spark job is initiated, executed, and managed in a cluster, including how components interact, programs are called, memory is allocated, and executors are assigned.

🔥 Apache Spark Job Execution – Complete Lifecycle & Architecture

📌 Step 0: Your Program – Spark Application Code

You write a PySpark or Scala/Java application like:

spark = SparkSession.builder.appName("MyJob").getOrCreate()
df = spark.read.csv("input.csv")
result = df.filter(...).groupBy(...).agg(...)
result.write.parquet("output/")

This script constitutes a Spark Application.

✅ Step 1: Driver Process Launch

• The driver is the master JVM process that initiates SparkContext and orchestrates the entire job.
• This is where the SparkSession lives.
• Driver responsibilities:
  • Reads your code
  • Builds DAG of transformations
  • Converts to physical plan
  • Schedules and coordinates executors

✅ Step 2: Cluster Manager Interaction

When you call .getOrCreate(), Spark communicates with a cluster manager:

The cluster manager allocates:

• 1 driver container/process
• N executor containers/processes

⚙️ Resource Allocation Parameters

--num-executors 4
--executor-memory 4G
--executor-cores 2

✅ Step 3: SparkContext + DAG Creation

Logical Plan

• Spark creates a DAG (Directed Acyclic Graph) of transformations (e.g., map, filter, groupBy).
• This is the logical plan: it represents what to do, not how to do it.

✅ Step 4: DAG Scheduler → Stage Creation

• Spark divides the DAG into stages based on shuffle boundaries.
  • Narrow transformations (map, filter): stay in same stage.
  • Wide transformations (groupBy, join): new stage, causes a shuffle.
• Each stage is broken into tasks — one task per partition.

Example:

df = spark.read.csv("...").filter(...).groupBy(...).count()

Stages:

• Stage 1: Read & Filter
• Stage 2: GroupBy (shuffle required)

✅ Step 5: TaskScheduler → Task Execution

The TaskScheduler sends tasks to executors.

🔹 Executors

• JVM processes launched on worker nodes.
• Run tasks assigned by the driver.
• Store data in memory/disk using block manager.
• Cache/persist RDDs/DataFrames.
• Communicate results back to driver.

Each executor has:

• N cores (executor-cores)
• M memory (executor-memory)

✅ Step 6: Execution Flow

1. Driver sends tasks to executors.
2. Executors run tasks in parallel (one per core).
3. Intermediate data is:
   • Held in memory (if caching)
   • Written to disk (if shuffling)
4. Shuffled data is pulled by downstream stages.

✅ Step 7: Memory Management

Spark divides executor memory:

In Spark 3.x (Unified Memory Manager), execution and storage share memory dynamically.

Great! Let’s now deep dive into Step 7: Memory Allocation per Executor in Spark, including how memory is calculated, what config values matter, and how dynamic memory sharing works in Spark’s Unified Memory Manager.

📦 Step 7: Memory Allocation per Executor (in Spark 3.x+)

🔥 What Is Executor Memory?

Each Spark executor runs on a worker node and has a JVM with its own memory space.

This memory is divided into:

✅ Key Configuration Parameters

🧮 How Spark Memory Is Calculated

Let’s assume:

spark.executor.memory = 4g

🔢 Breakdown:

And…

| Non-heap Overhead | spark.executor.memoryOverhead | ~384 MB (default) |

✅ Unified Memory Manager (Spark 2.0+)

Execution and storage memory are not fixed:

• If your job does no caching, storage memory is released to execution.
• If execution needs memory and storage is underused → it borrows it.
• If caching is needed and there’s space → storage can grow.
• Eviction happens if execution needs more and storage has cached data.

🔀 Dynamic Memory Sharing

Scenario A: Execution-intensive stage

• Sorts, joins, shuffles → use most of Unified Memory
• Spark evicts cached data if needed (LRU policy)

Scenario B: Heavy caching

• Many .cache() or .persist() calls
• Execution will fail if it can’t get enough memory (OOM)

✅ Memory Tuning Tips

🧠 Spark UI & Monitoring

• Executor tab: shows memory used for storage, execution, overhead.
• Storage tab: shows cached RDD/DataFrame sizes.
• Task tab: shows spills to disk (bad sign = not enough memory).

🧪 Example:

spark-submit \
  --executor-memory 4G \
  --executor-cores 4 \
  --conf spark.memory.fraction=0.6 \
  --conf spark.memory.storageFraction=0.5 \
  --conf spark.executor.memoryOverhead=512M \
  app.py

Memory layout:

4 GB executor heap
- 300 MB reserved
= 3.7 GB usable

0.6 × 3.7 = 2.2 GB Unified Memory
→ 1.1 GB Execution
→ 1.1 GB Storage
~ 1.5 GB User Memory
+ 512 MB memoryOverhead (off-heap)

🔚 Summary

Pages: 1 2 3 4 5 6 7 8