Apache Airflow Orchestration

Author, schedule, and monitor data workflows with Apache Airflow. Generates DAGs, custom operators, TaskFlow API pipelines, sensors, XCom patterns, and deferrable operators for production orchestration.

This skill writes idiomatic Airflow 2.x DAGs using the TaskFlow API, sets up dynamic task mapping for fan-out patterns, configures Celery/Kubernetes executors, manages connections and variables, and migrates legacy PythonOperator DAGs to the modern decorator syntax. Covers MWAA, Astronomer, and self-hosted deployments.

When to use

Use when building scheduled ETL pipelines, orchestrating multi-system data flows, migrating Airflow 1.x DAGs to 2.x, or troubleshooting stuck tasks and scheduler issues.

Examples

TaskFlow API pipeline

Modern decorator-based DAG with dependencies

Write an Airflow DAG using the TaskFlow API that extracts from Postgres, transforms with pandas, and loads into Snowflake — with retries, SLA alerts, and a Slack failure callback

Dynamic task mapping

Fan-out processing across S3 partitions

Build a dynamically-mapped Airflow task that processes each Parquet file in an S3 prefix in parallel, then aggregates results downstream