Spark and shuffles

Apache Spark generalizes MapReduce. Instead of two fixed steps, you write a whole pipeline, and Spark turns it into a graph of operators over partitioned data. Intermediate results can stay in memory, so chained steps stop writing to disk after every job. Most of the tuning work comes down to one thing: avoiding expensive shuffles.

The big idea

Think of a factory line. Most stations work on their own parts. Some stations need parts from everyone, so the parts must travel across the floor first. Those trips are the shuffle, and they are the slow part.

Spark splits the pipeline into stages at each shuffle. Steps inside a stage run together without moving data.

Walking through it

This sketch shows the shape of a typical Spark job. It reads from S3 paths that you would need to replace, so treat it as a sketch.

from pyspark.sql import functions as F

clicks = spark.read.parquet("s3://lake/clicks/date=2026-10-13/")
ads = spark.read.parquet("s3://lake/dim_ads/")

daily = (clicks
    .filter(F.col("is_bot") == False)            # narrow: no data movement
    .join(ads, "ad_id")                          # wide: shuffle, or broadcast if ads is small
    .groupBy("advertiser_id", "campaign_id")     # wide: shuffle
    .agg(F.count("*").alias("clicks"),
         F.countDistinct("user_id").alias("users")))

• Narrow steps such as filter, select and map process each partition alone. They run in one stage.
• Wide steps such as groupBy, join and distinct need rows from many partitions. Each one starts a new stage.

A broadcast join avoids the shuffle of the big table. The small table is copied to every executor. Spark does this automatically below a size threshold, which is 10 MB by default.

Skew is the other cost. If one key holds 30% of the rows and the data is split into 10 even partitions, the partition with that key gets 30% of the work. That is three times the average share of 10%. Salting the key, or letting adaptive query execution split the skewed partition, spreads it out.

Spark recovers from failures with lineage. It remembers how each partition was computed, and recomputes a lost one from its parents.

Common mistakes

• Treating every join as cheap. Check the plan for shuffles before you trust a job's runtime.
• Ignoring skew. One hot key can make one task run far longer than the rest.
• Assuming lineage protects side effects. Recomputing a partition re-runs any write it makes to an outside system.

Check yourself

1. Why does a filter not start a new stage but a groupBy does? A filter works on each partition alone. A groupBy needs rows from many partitions, so Spark must shuffle.
2. When does Spark use a broadcast join? When the small table is below the size threshold, which is 10 MB by default.

Key takeaways

• Spark runs a graph of operators and keeps intermediate data in memory where it can.
• Narrow steps stay in one stage. Wide steps force a shuffle.
• Shuffles usually dominate cost, through network transfer, spills and serialization.
• Broadcast joins avoid shuffling the large side when the other side is small.
• Skewed keys make one task do most of the work.

Further learning

Next, see how stream processing adds time, windows and lateness, in event time and watermarks.

For the batch picture, see batch and stream processing.