Skill UI

Guides engineers through data partitioner setups within ETL workflows, offering best practice advice, ready-to-run code, and validation for streaming and batch pipelines.

Automated Data Quality Checker offers guidance, best practices, and production-ready code for validating pipelines whenever you work with ETL, streaming, or data transformation scenarios.

This skill provides production patterns for implementing robust data quality validation using Great Expectations, dbt tests, and formal data contracts. It is essential for building reliable, automated, and verifiable data pipelines, ensuring data integrity throughout the entire ETL/ELT process, and establishing clear data governance rules.

This comprehensive workflow guides the entire database lifecycle, from initial architecture design (covering both SQL and NoSQL) to advanced implementation, optimization, and maintenance. It covers crucial stages like schema design, implementing migrations, optimizing query performance, developing robust data pipelines (using tools like Airflow and dbt), and managing data quality and operations. Ideal for engineers building complex, production-grade data systems.

This guide provides expert patterns for building robust, observable infrastructure for database migrations. It covers real-time monitoring using Change Data Capture (CDC), metrics collection (Prometheus/Grafana), and anomaly detection. Use this skill when setting up CDC pipelines (e.g., Debezium with Kafka) or implementing comprehensive observability for database schema changes and data synchronization processes.

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This comprehensive guide outlines the end-to-end process of building production-grade ETL data pipelines using Databricks and Delta Lake. It implements the robust Medallion Architecture (Bronze, Silver, Gold) for scalable data management. Techniques covered include Auto Loader for incremental streaming ingestion, MERGE INTO for upsert semantics, data cleansing, and advanced aggregation into analytics-ready layers. Ideal for establishing reliable data pipelines in a data lakehouse environment.

This guide outlines the complete MLOps workflow on Databricks. It covers feature engineering using Feature Store, tracking experiments with MLflow, managing model versions in the Model Registry, and deploying real-time inference endpoints via Mosaic AI Model Serving. Ideal for building robust, production-grade ML pipelines.

This skill provides a comprehensive guide on implementing robust data governance patterns within Delta Lake, leveraging Unity Catalog. It covers essential compliance topics such as data classification tagging, handling GDPR 'right-to-deletion' requests for PII, and establishing automated data retention policies. Use this when designing secure, compliant, and auditable data pipelines in a data lake environment.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

A comprehensive guide detailing the setup process for Databricks CLI v2 and Python SDK. It covers essential steps for connecting to various Databricks workspaces, including configuring Personal Access Tokens (PAT), interactive OAuth U2M flows, and secure OAuth M2M service principals for CI/CD pipelines. Essential for developers integrating Databricks into their projects.