Apache Arrow

Move and process columnar data at speed with Apache Arrow. Uses zero-copy in-memory format, Arrow Flight for fast transport, and interop across pandas, Polars, DuckDB, and Parquet without serialization overhead.

This skill makes Claude an Apache Arrow expert. It works with the columnar in-memory format that powers modern data tools, using zero-copy reads, compute kernels, and record batches for high-throughput processing. Covers Arrow Flight for cross-service data transfer, Parquet and Feather IO, and seamless handoff between pandas, Polars, DuckDB, and PyArrow with no serialization cost.

When to use

Use when optimizing data interchange between tools, working with Arrow tables or record batches, using Arrow Flight, or eliminating serialization overhead in a Python data pipeline.

Examples

Zero-copy handoff

pandas to DuckDB

Convert a pandas DataFrame to an Arrow table and query it in DuckDB with zero copy

Arrow Flight

Fast transport

Set up an Arrow Flight server to stream record batches between two Python services

Read Parquet fast

Columnar IO

Use PyArrow to read selected columns from a large Parquet file into an Arrow table