HintsToday

Let’s visualize how Spark schedules tasks when reading files (like CSV, Parquet, or from Hive), based on:

• File size
• File count
• Block size
• Partitions
• Cluster resources

⚙️ Step-by-Step: How Spark Schedules Tasks from Files

🔹 Step 1: Spark reads file metadata

When you call:

df = spark.read.csv("/data/large_file.csv")

• Spark queries the filesystem (HDFS/S3/DBFS) to get:
  • File sizes
  • Number of files
  • Block sizes (128MB in HDFS by default)
• Spark splits large files into logical input splits.

🔹 Step 2: Input Splits → Tasks

💡 Each input split = 1 Spark task in the first stage.

🔹 Step 3: Spark Schedules Tasks on Executors

• Spark sends 1 task per partition to an available executor core.
• If you have:
  • 4 executors × 4 cores each = 16 parallel tasks
  • But 32 splits → runs in 2 waves

🧠 Example: 8 Input Splits on 2 Executors (4 cores each)

Stage 0:
[Task 0] → Executor 1 Core 1
[Task 1] → Executor 1 Core 2
[Task 2] → Executor 1 Core 3
[Task 3] → Executor 1 Core 4
[Task 4] → Executor 2 Core 1
[Task 5] → Executor 2 Core 2
[Task 6] → Executor 2 Core 3
[Task 7] → Executor 2 Core 4

Once all tasks finish, Stage 0 completes.

🔁 Step 4: Shuffle Creates More Tasks

If a transformation like groupBy() or join() is done:

df.groupBy("col").count()

• Spark creates a shuffle stage
• Partitions into spark.sql.shuffle.partitions (default = 200)
• Each shuffle partition → 1 task in next stage

🔍 Key Concepts

📊 Visual Diagram

          CSV File (1 GB)
        ─────────────────────
        | Split 0: 0–128MB   | → Task 0 → Executor 1 Core 1
        | Split 1: 128–256MB | → Task 1 → Executor 2 Core 2
        | Split 2: 256–384MB | → Task 2 → Executor 1 Core 2
        ...
        | Split 7: 896–1024MB| → Task 7 → Executor 2 Core 4

✅ Real Spark UI Checkpoints

In the Spark UI → Stages Tab:

• See number of tasks per stage
• Hover over each task to view input split and data read

🔚 Summary Table

Pages: 1 2