Case Study 02 — The Long Weekend: Reading Spark End to End
01An end-to-end Apache Spark learning and performance-tuning case study for Northwind Traders, following a Lisbon sales report from lazy evaluation through query planning, joins, and data skipping.
Challenge
Northwind Traders' coastal markets division needed its high-volume August festival-weekend sales report ready by 06:00. Engineer Rui Almeida set out to understand how Spark defers work, plans execution, moves data across a cluster, and reads less—so the morning market report could remain trustworthy as data volume surged.
Solution
- Explains Parquet columnar storage and Delta transaction-log guarantees, then traces lazy transformations to actions, DAG stages, tasks, and the narrow-versus-wide distinction.
- Uses Catalyst plans, explain() output, and query profiles to identify FileScan, pushed filters, Project, Exchange shuffles, skew, and Photon execution on serverless compute.
- Compares broadcast joins for small kiosk dimensions against sort-merge joins, and shows how early date predicates and selected columns reduce data read from Parquet.
- Covers region partition pruning, flexible liquid clustering, and AQE coalescing partitions, switching joins, and splitting skew; includes a 20-operation narrow/wide exercise and a before-and-after report tuning assignment.
- Spark topics covered
- 4
- Transformations classified
- 20
- Morning report deadline
- 06:00
- Databricks
- Apache Spark
- PySpark
- Delta Lake
- Parquet
- Catalyst Optimizer
- Photon
- AQE
- Liquid Clustering
