Skill UI

Automates the creation of Airflow DAGs for ETL and streaming data workflows, offering best-practice guidance, production-ready code samples, and validation against common pipeline standards.

A comprehensive guide to building production-ready Apache Airflow Directed Acyclic Graphs (DAGs). This skill covers best practices for operators, sensors, testing, deployment, and designing robust data pipelines. Use this when orchestrating complex workflows, managing ETL processes, or ensuring reliable batch job execution in a production environment.

Guides you through building Airflow operators for data pipelines with automated best practices, production-ready code/config generation, and validation for ETL, transformation, orchestration, and streaming scenarios.

Automates change-data-capture pipeline planning by guiding developers through ETL, orchestration, and streaming best practices, then generating validated configs and code when “cdc pipeline creator” is requested.

Automates beam pipeline builder tasks by guiding best practices, generating production-ready code, and validating ETL, transformation, orchestration, and streaming workflows.

A comprehensive guide for deploying applications authenticated with Clerk to various major hosting platforms, including Vercel, Netlify, Railway, and custom Docker setups. It details environment variable setup, domain configuration, webhook handling, and provides post-deployment verification scripts to ensure a robust production environment.

Guides users through compression optimizer scenarios within data pipelines, offering best practices, code templates, and validation steps for ETL, transformation, and streaming workflows.

Guides you through building Dagster data pipelines with best-practice advice, ready-to-use configurations, and validation checks for ETL, transformation, orchestration, and streaming workflows whenever the Dagster pipeline creator skill is activated.

Automates guidance for data catalog updater tasks, covering ETL, transformation, orchestration, and streaming workflows to keep pipeline metadata accurate.

Expert in designing, building, and optimizing scalable data pipelines, modern data warehouses, and real-time streaming architectures. Mastery covers the complete modern data stack, including Spark, dbt, Airflow, and cloud services (AWS, Azure, GCP). Focuses on ELT/ETL, data lakehouse implementation (Delta Lake, Iceberg), data governance, and ensuring high data quality and operational reliability.

Guides data engineering patterns for ETL pipelines, Spark analytics, and quality checks, showcasing Python and SQL snippets to extract, transform, load, validate, and model sales data in modern warehouses.

An expert in designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data. Covers advanced topics including ETL/ELT patterns, Kappa/Lambda architectures, workflow orchestration (Airflow/Prefect), transformation using dbt/Spark, and managing modern storage formats like Delta Lake and Iceberg. Focuses heavily on data quality, monitoring, and cost optimization strategies.