Parquet Columnar Storage
Work with Apache Parquet effectively. Chooses compression codecs, designs partition layouts, applies predicate and column pushdown, and handles schema evolution so analytical reads stay fast and cheap.
This skill covers columnar Parquet patterns for data lakes and analytics. It writes with the right compression (snappy, gzip, zstd), designs Hive-style partition layouts, uses predicate pushdown and column projection to skip data, evolves schemas safely, and reads efficiently from pandas, PyArrow, Polars, and DuckDB.
parquet columnar partitioning data-lake storage
When to use
Use when writing or reading Parquet files, choosing compression, designing partitions, applying predicate/column pushdown, or handling Parquet schema evolution.
Examples
Partitioned write
Lay out a Parquet dataset
Write my events DataFrame to a partitioned Parquet dataset by year and month with zstd compression
Predicate pushdown
Read only what you need
Read only the amount and country columns from this Parquet dataset for country = 'US' using predicate and column pushdown
Schema evolution
Add a column safely
I added a new nullable column to my Parquet dataset. Show me how to read old and new files together without breaking