Apache Spark & PySpark

Process big data at scale with Apache Spark. Writes PySpark ETL jobs, builds DataFrame and Spark SQL transformations, tunes shuffles, partitions, joins, and caching, fixes data skew, and runs on Databricks, EMR, Dataproc, or Kubernetes.

This skill guides distributed data processing with Apache Spark and PySpark. It writes batch and streaming ETL jobs, builds DataFrame and Spark SQL transformations, tunes partitioning, shuffles, broadcast joins, and caching, diagnoses and fixes data skew and out-of-memory errors, and deploys to standalone, YARN, Kubernetes, or managed platforms like Databricks, EMR, and Dataproc.

When to use

Use when writing PySpark or Spark SQL jobs, building distributed ETL, tuning a slow Spark job, fixing data skew or shuffles, or deploying Spark on Databricks, EMR, Dataproc, or Kubernetes.

Examples

Build a PySpark ETL job

Distributed transform to Parquet

Write a PySpark job that reads raw JSON events from S3, cleans and dedupes them, and writes partitioned Parquet to a curated bucket

Fix a slow Spark job

Tune shuffles and skew

This Spark join is shuffling terabytes and one task runs forever. Diagnose the skew and rewrite it with a broadcast join and salting

Spark SQL aggregation

Window functions at scale

Write Spark SQL that computes 7-day rolling active users from an events DataFrame using window functions