Skill UI

This skill provides production patterns for implementing robust data quality validation using Great Expectations, dbt tests, and formal data contracts. It is essential for building reliable, automated, and verifiable data pipelines, ensuring data integrity throughout the entire ETL/ELT process, and establishing clear data governance rules.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

This advanced skill orchestrates a complete, multi-stage research idea discovery pipeline. It systematically guides the user from a broad research direction to a fully validated, actionable project proposal. The workflow integrates literature surveying, creative brainstorming, novelty checking, and critical review, culminating in a detailed idea report, refined proposal, and experimental plan for implementation.

This skill systematically verifies the novelty of a proposed research method or idea against vast global literature. It identifies core technical claims, performs multi-source searches across arXiv, Google Scholar, and major conferences (ICLR, NeurIPS, ICML), and utilizes advanced LLMs for cross-model verification. The output is a structured report, providing a novelty score, identifying the closest prior work, and offering a strategic recommendation to guide the research direction.

A comprehensive guide for executing core Snowflake data loading workflows. This covers setting up stages (S3, GCS, Azure), defining file formats (CSV, JSON, Parquet), and using COPY INTO for bulk loading. It also details advanced continuous ingestion using Snowpipe, enabling real-time data synchronization from cloud storage into Snowflake tables.

Master the creation of robust ELT (Extract, Load, Transform) data pipelines within Snowflake. This guide demonstrates advanced data orchestration using Snowflake Streams for Change Data Capture (CDC), Tasks for scheduled transformations and DAG management, and Dynamic Tables for declarative, auto-refreshing data models. Ideal for building sophisticated, scalable data warehousing solutions.