Skill UI

This comprehensive guide outlines the end-to-end process of building production-grade ETL data pipelines using Databricks and Delta Lake. It implements the robust Medallion Architecture (Bronze, Silver, Gold) for scalable data management. Techniques covered include Auto Loader for incremental streaming ingestion, MERGE INTO for upsert semantics, data cleansing, and advanced aggregation into analytics-ready layers. Ideal for establishing reliable data pipelines in a data lakehouse environment.

This guide outlines the complete MLOps workflow on Databricks. It covers feature engineering using Feature Store, tracking experiments with MLflow, managing model versions in the Model Registry, and deploying real-time inference endpoints via Mosaic AI Model Serving. Ideal for building robust, production-grade ML pipelines.

This skill provides a comprehensive guide on implementing robust data governance patterns within Delta Lake, leveraging Unity Catalog. It covers essential compliance topics such as data classification tagging, handling GDPR 'right-to-deletion' requests for PII, and establishing automated data retention policies. Use this when designing secure, compliant, and auditable data pipelines in a data lake environment.

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

A comprehensive guide detailing the setup process for Databricks CLI v2 and Python SDK. It covers essential steps for connecting to various Databricks workspaces, including configuring Personal Access Tokens (PAT), interactive OAuth U2M flows, and secure OAuth M2M service principals for CI/CD pipelines. Essential for developers integrating Databricks into their projects.

This skill guides the setup and management of Databricks resources across development, staging, and production environments. It covers best practices for using isolated workspaces, managing environment-specific secrets, implementing Asset Bundles, and integrating with CI/CD pipelines to ensure consistent and reliable data deployments.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

Guides you through generating dbt models inside data pipelines, covering setup, code generation, configuration, and validation while following best practices for ETL, orchestration, and streaming workloads.

Automates dbt test creation within data pipelines, offering guided best practices, code generation, and validation to keep ETL, transformation, and streaming workflows reliable.

This skill manages and refines search results obtained from Exa's neural search API. It provides comprehensive control over content extraction scope, allowing users to selectively retrieve metadata, highlights, full text, or structured summaries. Key features include implementing result caching with TTL, deduplicating citations, and crucial token budget management, ensuring the retrieved context fits within LLM context window limits for robust RAG pipelines.

This skill provides guidance and patterns for generating structured JSON files that define complex workflow pipelines. It is essential for chaining multiple distinct AI models together, enabling multi-step, automated data processing. Use this when a single task requires a sequence of interconnected AI operations or services.