Skill UI

An expert in designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data. Covers advanced topics including ETL/ELT patterns, Kappa/Lambda architectures, workflow orchestration (Airflow/Prefect), transformation using dbt/Spark, and managing modern storage formats like Delta Lake and Iceberg. Focuses heavily on data quality, monitoring, and cost optimization strategies.

Offers automated guidance for implementing data lineage trackers across ETL and streaming pipelines, including best practices on transformations, orchestration, and validation for production environments.

Guides engineers through data partitioner setups within ETL workflows, offering best practice advice, ready-to-run code, and validation for streaming and batch pipelines.

Automated Data Quality Checker offers guidance, best practices, and production-ready code for validating pipelines whenever you work with ETL, streaming, or data transformation scenarios.

This skill provides production patterns for implementing robust data quality validation using Great Expectations, dbt tests, and formal data contracts. It is essential for building reliable, automated, and verifiable data pipelines, ensuring data integrity throughout the entire ETL/ELT process, and establishing clear data governance rules.

Automates the data story outlining process for analytic workflows, offering guidance on SQL, visualization, statistics, and BI so teams can structure insights consistently.

This comprehensive workflow guides the entire database lifecycle, from initial architecture design (covering both SQL and NoSQL) to advanced implementation, optimization, and maintenance. It covers crucial stages like schema design, implementing migrations, optimizing query performance, developing robust data pipelines (using tools like Airflow and dbt), and managing data quality and operations. Ideal for engineers building complex, production-grade data systems.

This guide provides expert patterns for building robust, observable infrastructure for database migrations. It covers real-time monitoring using Change Data Capture (CDC), metrics collection (Prometheus/Grafana), and anomaly detection. Use this skill when setting up CDC pipelines (e.g., Debezium with Kafka) or implementing comprehensive observability for database schema changes and data synchronization processes.

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This comprehensive guide outlines the end-to-end process of building production-grade ETL data pipelines using Databricks and Delta Lake. It implements the robust Medallion Architecture (Bronze, Silver, Gold) for scalable data management. Techniques covered include Auto Loader for incremental streaming ingestion, MERGE INTO for upsert semantics, data cleansing, and advanced aggregation into analytics-ready layers. Ideal for establishing reliable data pipelines in a data lakehouse environment.

This guide outlines the complete MLOps workflow on Databricks. It covers feature engineering using Feature Store, tracking experiments with MLflow, managing model versions in the Model Registry, and deploying real-time inference endpoints via Mosaic AI Model Serving. Ideal for building robust, production-grade ML pipelines.

This skill provides a comprehensive guide on implementing robust data governance patterns within Delta Lake, leveraging Unity Catalog. It covers essential compliance topics such as data classification tagging, handling GDPR 'right-to-deletion' requests for PII, and establishing automated data retention policies. Use this when designing secure, compliant, and auditable data pipelines in a data lake environment.