Skill UI

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive guide outlines the end-to-end process of building production-grade ETL data pipelines using Databricks and Delta Lake. It implements the robust Medallion Architecture (Bronze, Silver, Gold) for scalable data management. Techniques covered include Auto Loader for incremental streaming ingestion, MERGE INTO for upsert semantics, data cleansing, and advanced aggregation into analytics-ready layers. Ideal for establishing reliable data pipelines in a data lakehouse environment.

This guide outlines the complete MLOps workflow on Databricks. It covers feature engineering using Feature Store, tracking experiments with MLflow, managing model versions in the Model Registry, and deploying real-time inference endpoints via Mosaic AI Model Serving. Ideal for building robust, production-grade ML pipelines.

This guide provides a comprehensive framework for minimizing cloud expenditure on Databricks. It covers identifying cost drivers via system billing tables, implementing cluster policies, utilizing spot instances for batch processing, right-sizing SQL warehouses, and automating the shutdown of idle development clusters. Use it when managing cloud spending or optimizing data platform usage.

This skill provides a comprehensive guide on implementing robust data governance patterns within Delta Lake, leveraging Unity Catalog. It covers essential compliance topics such as data classification tagging, handling GDPR 'right-to-deletion' requests for PII, and establishing automated data retention policies. Use this when designing secure, compliant, and auditable data pipelines in a data lake environment.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

A comprehensive guide for establishing robust, enterprise-grade access control within Databricks. Covers implementing Single Sign-On (SSO) via SCIM, leveraging Unity Catalog for fine-grained Role-Based Access Control (RBAC), defining cluster policies, and enforcing data governance measures like Row-Level Security (RLS) and column masking. Essential for large-scale, secure data platforms.

A comprehensive guide to setting up a minimal working example on Databricks. This skill covers creating a development cluster, uploading and executing notebooks (Python SDK), running tasks as jobs, and setting up a Serverless SQL Warehouse. It is ideal for new users or initializing a basic data engineering project.

A comprehensive runbook designed for on-call engineers to manage Databricks incidents, outages, and major job failures. It provides structured procedures for immediate triage, decision tree guidance (covering cluster failures, code errors, and data quality issues), and steps for evidence collection and postmortem documentation, ensuring rapid and systematic incident resolution.

A comprehensive guide detailing the setup process for Databricks CLI v2 and Python SDK. It covers essential steps for connecting to various Databricks workspaces, including configuring Personal Access Tokens (PAT), interactive OAuth U2M flows, and secure OAuth M2M service principals for CI/CD pipelines. Essential for developers integrating Databricks into their projects.