Datum Quantum
Databricks logoDatabricks

Portfolio / Databricks

Databricks projects

Databricks solutions spanning lakehouse engineering, streaming, governance, machine learning, and agentic AI.

Case Study 02 — The Long Weekend: Reading Spark End to End

01

An end-to-end Apache Spark learning and performance-tuning case study for Northwind Traders, following a Lisbon sales report from lazy evaluation through query planning, joins, and data skipping.

Challenge

Northwind Traders' coastal markets division needed its high-volume August festival-weekend sales report ready by 06:00. Engineer Rui Almeida set out to understand how Spark defers work, plans execution, moves data across a cluster, and reads less—so the morning market report could remain trustworthy as data volume surged.

Solution

  • Explains Parquet columnar storage and Delta transaction-log guarantees, then traces lazy transformations to actions, DAG stages, tasks, and the narrow-versus-wide distinction.
  • Uses Catalyst plans, explain() output, and query profiles to identify FileScan, pushed filters, Project, Exchange shuffles, skew, and Photon execution on serverless compute.
  • Compares broadcast joins for small kiosk dimensions against sort-merge joins, and shows how early date predicates and selected columns reduce data read from Parquet.
  • Covers region partition pruning, flexible liquid clustering, and AQE coalescing partitions, switching joins, and splitting skew; includes a 20-operation narrow/wide exercise and a before-and-after report tuning assignment.
Spark topics covered
4
Transformations classified
20
Morning report deadline
06:00
  • Databricks
  • Apache Spark
  • PySpark
  • Delta Lake
  • Parquet
  • Catalyst Optimizer
  • Photon
  • AQE
  • Liquid Clustering
View on GitHub(opens in a new tab)

Case Study 01 — The Smart Pricing Pilot

02

A Databricks Bronze-to-Silver billing pipeline for GreenGrid Solutions that turns noisy smart-meter telemetry into accurate, privacy-safe, audit-ready customer bills.

Challenge

GreenGrid's 100-home pilot had 62 unclean December readings, including negative consumption and phantom spikes up to 9,000 kWh. Spreadsheet billing applied peak rates inconsistently, leaking about $12K per month per 100 homes, while plain-text customer emails were copied across analyst desktops. The Series B review required a trustworthy design before scaling to 50,000 nodes.

Solution

  • Two ordered PySpark notebooks split out-of-range readings into quarantine_readings with a rejection_reason before any billing logic runs; the valid path accepts only kWh greater than 0 and less than 100.
  • The Silver pipeline joins meter readings to pricing plans, applies a configurable peak_multiplier of 2.0 from 18:00 through 20:59, and calculates FinalRate and Bill_Amount deterministically.
  • Customer email is converted to a SHA-256 CustomerHash and the raw email column is dropped before writing Silver_SmartBill, preserving joinable pseudonymous identity without storing plaintext PII.
  • Completeness and integrity checks reconcile all 12,474 December readings: 62 quarantined and 12,412 billed. Validation confirms the 19:30 peak example bills $1.50 and that Silver contains no raw email.
Readings reconciled
12,474
Invalid readings quarantined
62
Nodes targeted by Q3
50K
  • Databricks
  • PySpark
  • Delta Lake
  • Bronze / Silver
  • Data Quality Gates
  • Quarantine
  • SHA-256 Pseudonymization
  • Parameterized Widgets
View on GitHub(opens in a new tab)