Skill UI

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

Performs rigorous, zero-context verification of all quantitative claims, comparisons, and scope statements reported in a research paper against the raw evidence files. It is designed to detect confirmation bias, such as rounding errors, data cherry-picking, or misreported deltas, ensuring strict fidelity between published claims and underlying raw results. Ideal for pre-submission quality control.

Run arbitrary PySpark or pure Python code directly on Fabric Spark compute using the Livy API. These sessions are ephemeral, meaning no notebook artifacts are created or persisted, making it ideal for automated ETL processes, programmatic data transformations, or agent-driven compute workflows. Provides full read/write access to lakehouse Delta tables via Spark SQL.

This skill enables querying Delta Lake tables and raw files stored in OneLake using DuckDB. It supports two primary modes: local execution via CLI/Python for ad-hoc analysis, and in-notebook execution within Fabric, combining DuckDB's speed with notebook context. Ideal for data exploration, schema discovery, and data quality validation across the Fabric ecosystem.