Skill UI

Automated Data Quality Checker offers guidance, best practices, and production-ready code for validating pipelines whenever you work with ETL, streaming, or data transformation scenarios.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

A comprehensive guide to implementing best practices for security within Databricks. Learn how to manage secrets using scoped storage, enforce the principle of least privilege via Unity Catalog grants, audit API tokens, and apply advanced data protection techniques like column-level masking and row-level security for enhanced data governance and compliance.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

Guides you through generating dbt models inside data pipelines, covering setup, code generation, configuration, and validation while following best practices for ETL, orchestration, and streaming workloads.

Automates dbt test creation within data pipelines, offering guided best practices, code generation, and validation to keep ETL, transformation, and streaming workflows reliable.

Automated guidance for configuring and applying early stopping callbacks across ML training workflows, covering best practices, code generation, and validation for PyTorch, TensorFlow, and scikit-learn experiments.

Automates guidance for setting up and validating feature store connectors within ML deployment pipelines, covering best practices, code generation, and production monitoring for inference.

Automated assistant for forecast generator tasks in data analytics, guiding through SQL, visualization, statistics, and BI workflows with best practices, ready-to-run code, and validation.

Automates funnel analysis builder workflows by offering guidance on SQL, visualization, statistics, and BI best practices for data analytics teams needing production-ready insights.

Guides data-engineering teams through configuring incremental loads in pipelines, covering best practices, orchestration tips, example code, and output validation for streaming or batch workflows.