Dask Parallel Python

Scale Python and pandas with Dask. Parallelizes DataFrame and array workloads across cores or a cluster, tunes partitions and the task graph, handles larger-than-memory data, and diagnoses spill and shuffle bottlenecks from the dashboard.

This skill scales Python workloads with Dask: parallelizing pandas-style DataFrame and NumPy-style array operations across cores or a distributed cluster, choosing partition sizes and tuning the task graph, processing larger-than-memory datasets with lazy evaluation, and diagnosing memory spill, shuffle, and scheduler bottlenecks using the Dask dashboard.

When to use

Use when converting a pandas or NumPy job to Dask, parallelizing Python across cores or a cluster, processing larger-than-memory data, or debugging Dask memory spill and shuffles.

Examples

Scale a pandas job

Convert to Dask

Convert this pandas pipeline that runs out of memory into Dask DataFrame code that processes the data in partitions

Tune partitions

Right-size chunks

My Dask job reads a 50GB CSV and is slow. Recommend partition sizes and how to repartition for a groupby-aggregate

Debug memory spill

Read the dashboard

My Dask workers keep spilling to disk and dying. Help me read the dashboard and fix the memory pressure