Apache Airflow DAG Patterns
Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
Use this skill when
- Creating data pipeline orchestration with Airflow
- Designing DAG structures and dependencies
- Implementing custom operators and sensors
- Testing Airflow DAGs locally
- Setting up Airflow in production
- Debugging failed DAG runs
Do not use this skill when
- You only need a simple cron job or shell script
- Airflow is not part of the tooling stack
- The task is unrelated to workflow orchestration
Instructions
- Identify data sources, schedules, and dependencies.
- Design idempotent tasks with clear ownership and retries.
- Implement DAGs with observability and alerting hooks.
- Validate in staging and document operational runbooks.
Refer to resources/implementation-playbook.md for detailed patterns, checklists, and templates.
Safety
- Avoid changing production DAG schedules without approval.
- Test backfills and retries carefully to prevent data duplication.
Resources
-
resources/implementation-playbook.md for detailed patterns, checklists, and templates.
