Skill UI

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

A comprehensive runbook designed for on-call engineers to manage Databricks incidents, outages, and major job failures. It provides structured procedures for immediate triage, decision tree guidance (covering cluster failures, code errors, and data quality issues), and steps for evidence collection and postmortem documentation, ensuring rapid and systematic incident resolution.

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

Automates dataset loader creation within ML Training workflows, offering best-practice guidance, validation checks, and production-ready code to accelerate data prep and model tuning.

Automates guidance for date range analytics, covering SQL, visualization, statistics, and BI, with code generation and validation to support data analytics workflows.

This guide provides essential best practices for building reliable, fault-tolerant applications in Go using DBOS durable workflows. It covers architectural patterns for integrating DBOS into existing Go code, defining complex workflows, managing steps, and utilizing queues for robust concurrency control. Essential reading for developers setting up, configuring, or testing mission-critical asynchronous processes.

A comprehensive guide detailing best practices for building reliable, fault-tolerant Python applications utilizing DBOS durable workflows. This resource is essential when integrating DBOS into existing Python codebases, defining complex workflows and steps, managing concurrency via queues, or implementing robust workflow communication patterns.

A comprehensive guide for building reliable, fault-tolerant applications using DBOS durable workflows within a TypeScript environment. It covers best practices for initialization, structuring complex workflows and steps, managing concurrency via queues, and integrating DBOS into existing TypeScript codebases.