Skill UI

Actionable guidance for analyzing plans, indexing, connection pooling, partitioning, and detecting N+1 queries to keep PostgreSQL and MySQL workloads fast and reliable.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

A comprehensive guide to setting up a minimal working example on Databricks. This skill covers creating a development cluster, uploading and executing notebooks (Python SDK), running tasks as jobs, and setting up a Serverless SQL Warehouse. It is ideal for new users or initializing a basic data engineering project.

A comprehensive runbook designed for on-call engineers to manage Databricks incidents, outages, and major job failures. It provides structured procedures for immediate triage, decision tree guidance (covering cluster failures, code errors, and data quality issues), and steps for evidence collection and postmortem documentation, ensuring rapid and systematic incident resolution.

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

This skill provides comprehensive observability for the Databricks platform. It enables users to monitor job run histories, track cluster utilization and associated costs (DBU), detect slow SQL queries, and audit access logs. Use it for setting up robust monitoring dashboards, ensuring pipeline health, and optimizing resource consumption across the entire data stack.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

Automates Datadog monitoring workflows via Composio’s Rube MCP, letting you query metrics, search logs, manage monitors and dashboards, and create events or downtimes after connecting to the MCP toolkit and tools list.

Guides using Rube MCP’s Composio toolkit to discover current Datagma tools, verify connections, and execute workflows while adhering to live schemas, session reuse, and required memory parameters.