Skill UI

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

A comprehensive guide and implementation pattern for managing rate limits, throttling, and ensuring idempotency when interacting with the Databricks API. This skill demonstrates techniques like exponential backoff with jitter, token-bucket rate limiting, and using idempotency tokens to make bulk and concurrent API calls reliable and efficient, minimizing HTTP 429 errors.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

A comprehensive guide to production-ready patterns for the Databricks Python SDK. This skill covers essential best practices including singleton client initialization, structured error handling with typed exceptions, context-managed cluster lifecycle control (ensuring cleanup), and implementing type-safe job definitions. It is essential for building robust, maintainable, and idiomatic integrations with the Databricks platform.

A comprehensive guide to implementing best practices for security within Databricks. Learn how to manage secrets using scoped storage, enforce the principle of least privilege via Unity Catalog grants, audit API tokens, and apply advanced data protection techniques like column-level masking and row-level security for enhanced data governance and compliance.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

Automates Datadog monitoring workflows via Composio’s Rube MCP, letting you query metrics, search logs, manage monitors and dashboards, and create events or downtimes after connecting to the MCP toolkit and tools list.

Guides using Rube MCP’s Composio toolkit to discover current Datagma tools, verify connections, and execute workflows while adhering to live schemas, session reuse, and required memory parameters.

Automate Datarobot operations via Composio's Rube MCP by discovering tool schemas, managing connections, and executing workflows with RUBE_MULTI_EXECUTE_TOOL while observing schema compliance and session reuse best practices.

Automates dataset loader creation within ML Training workflows, offering best-practice guidance, validation checks, and production-ready code to accelerate data prep and model tuning.

Automates guidance for date range analytics, covering SQL, visualization, statistics, and BI, with code generation and validation to support data analytics workflows.