Apache Iceberg Lakehouse

Build an open lakehouse on Apache Iceberg. Designs Iceberg tables and partitions, manages schema and partition evolution, handles snapshots, time travel, and compaction, and queries from Spark, Trino, and DuckDB across S3 or GCS.

This skill builds and operates an open lakehouse on Apache Iceberg: designing tables with hidden partitioning, evolving schema and partition specs without rewriting data, using snapshots for time travel and rollback, compacting small files and expiring snapshots for performance, choosing a catalog (REST, Glue, Nessie), and querying the same tables from Spark, Trino, Flink, and DuckDB on S3 or GCS.

When to use

Use when designing Apache Iceberg tables, evolving schema or partitions, managing snapshots, time travel, or compaction, or querying an Iceberg lakehouse from Spark, Trino, or DuckDB.

Examples

Create an Iceberg table

Hidden partitioning

Create an Apache Iceberg table for events partitioned by day using hidden partitioning, stored as Parquet on S3

Evolve without rewrites

Schema and partition change

Add a column and change the partition spec on my Iceberg table without rewriting the existing data

Compaction and cleanup

Keep it fast

Write the Iceberg maintenance to compact small files and expire snapshots older than 7 days