Skill UI

This skill provides production patterns for implementing robust data quality validation using Great Expectations, dbt tests, and formal data contracts. It is essential for building reliable, automated, and verifiable data pipelines, ensuring data integrity throughout the entire ETL/ELT process, and establishing clear data governance rules.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

This advanced skill systematically verifies the novelty of a proposed research method or technical idea against the global body of academic literature. It guides the user to extract core technical claims, performs multi-source searches across major academic databases (e.g., arXiv, Google Scholar, top conferences), and utilizes high-level LLM reasoning to generate a detailed, structured novelty assessment report. Ideal for researchers seeking to validate the unique contribution of their work.

This skill performs a zero-context verification audit, ensuring that every quantitative claim, comparison, and scope statement within a research paper precisely matches the provided raw result files. It is designed to eliminate confirmation bias, catching discrepancies such as incorrect rounding, cherry-picked best seeds, or misreported performance deltas before publication. Ideal for validating paper-to-evidence fidelity.

A comprehensive guide for executing core Snowflake data loading workflows. This covers setting up stages (S3, GCS, Azure), defining file formats (CSV, JSON, Parquet), and using COPY INTO for bulk loading. It also details advanced continuous ingestion using Snowpipe, enabling real-time data synchronization from cloud storage into Snowflake tables.

Master the creation of robust ELT (Extract, Load, Transform) data pipelines within Snowflake. This guide demonstrates advanced data orchestration using Snowflake Streams for Change Data Capture (CDC), Tasks for scheduled transformations and DAG management, and Dynamic Tables for declarative, auto-refreshing data models. Ideal for building sophisticated, scalable data warehousing solutions.

Execute arbitrary PySpark or pure Python code directly on Fabric Spark compute using the Livy API. This method ensures that no notebook artifacts are created or persisted, making it ideal for automated, ephemeral data transformations and ETL pipelines. It provides full read/write access to Delta lakehouse tables.