Skill UI

Automates the creation of Airflow DAGs for ETL and streaming data workflows, offering best-practice guidance, production-ready code samples, and validation against common pipeline standards.

A comprehensive guide to building production-ready Apache Airflow Directed Acyclic Graphs (DAGs). This skill covers best practices for operators, sensors, testing, deployment, and designing robust data pipelines. Use this when orchestrating complex workflows, managing ETL processes, or ensuring reliable batch job execution in a production environment.

Guides you through building Airflow operators for data pipelines with automated best practices, production-ready code/config generation, and validation for ETL, transformation, orchestration, and streaming scenarios.

Expert in designing, building, and optimizing scalable data pipelines, modern data warehouses, and real-time streaming architectures. Mastery covers the complete modern data stack, including Spark, dbt, Airflow, and cloud services (AWS, Azure, GCP). Focuses on ELT/ETL, data lakehouse implementation (Delta Lake, Iceberg), data governance, and ensuring high data quality and operational reliability.

An expert in designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data. Covers advanced topics including ETL/ELT patterns, Kappa/Lambda architectures, workflow orchestration (Airflow/Prefect), transformation using dbt/Spark, and managing modern storage formats like Delta Lake and Iceberg. Focuses heavily on data quality, monitoring, and cost optimization strategies.

This comprehensive workflow guides the entire database lifecycle, from initial architecture design (covering both SQL and NoSQL) to advanced implementation, optimization, and maintenance. It covers crucial stages like schema design, implementing migrations, optimizing query performance, developing robust data pipelines (using tools like Airflow and dbt), and managing data quality and operations. Ideal for engineers building complex, production-grade data systems.

Mastering the full lifecycle of machine learning models from data ingestion to deployment. This expertise covers building robust, production-grade MLOps pipelines using orchestrators like Kubeflow, Airflow, or Prefect. Key skills include feature store implementation with Feast, rigorous data validation, experiment tracking (e.g., MLflow), model registry management, and automated deployment workflows. Essential for automating model training and ensuring reproducibility at scale.

This skill provides comprehensive guidance for building production-grade MLOps pipelines. It covers the entire machine learning lifecycle, from initial data ingestion and rigorous feature engineering to model training, advanced validation (A/B testing, performance drift), and robust deployment strategies (Canary, Blue-green). It enables the design and orchestration of scalable, reproducible, and observable ML systems using tools like Airflow and Kubeflow.

This comprehensive guide covers the entire MLOps lifecycle, specializing in building robust, scalable, and production-grade Machine Learning systems. Expertise includes orchestrating complex ML pipelines using Kubeflow, Airflow, Prefect, and cloud-native services (AWS SageMaker, Azure ML, GCP Vertex AI). It covers experiment tracking (MLflow, W&B), model versioning, model registries, infrastructure as code (Terraform), and container deployment using Kubernetes. Ideal for engineers tasked with moving ML models from research to reliable, managed production environments.

This comprehensive guide details the expertise needed for a Senior Data Engineer. It covers building, optimizing, and governing scalable data pipelines and robust ETL/ELT systems using modern tools like Spark, Kafka, Airflow, and dbt. Includes decision frameworks for architecting data solutions (Batch vs Streaming, Lakehouse vs Warehouse) and implementing advanced data modeling patterns.

This skill provides the core mechanism for integrating Hex into external data pipelines. It enables triggering, monitoring, and managing Hex projects (e.g., via Airflow or Dagster) programmatically. Users can implement parameterized runs, execute multi-step ETL workflows sequentially, and handle status polling and error management, making it ideal for production data orchestration.

This guide provides production-ready patterns for integrating with the Hex platform API using TypeScript and Python. It covers essential best practices like robust retry logic, typed client usage, pipeline orchestration, and Airflow integration, helping developers establish standardized, reliable coding standards for complex Hex workflows.