Skill UI

Offers automated guidance for implementing data lineage trackers across ETL and streaming pipelines, including best practices on transformations, orchestration, and validation for production environments.

This skill provides production patterns for implementing robust data quality validation using Great Expectations, dbt tests, and formal data contracts. It is essential for building reliable, automated, and verifiable data pipelines, ensuring data integrity throughout the entire ETL/ELT process, and establishing clear data governance rules.

This skill models the expertise of a senior data scientist, covering the entire data science lifecycle. It provides advanced capabilities in statistical modeling, machine learning (deep learning, XGBoost), predictive analytics, and business intelligence. From raw data exploration (EDA) to deploying production-ready models and creating actionable dashboards, it covers technical depth across Python, R, SQL, and cloud platforms, while applying domain knowledge in finance, marketing, and operations.

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

A comprehensive tool for querying and analyzing deep blockchain data across multiple EVM-compatible networks, including Ethereum, Polygon, Arbitrum, and Optimism. It supports tracking transactions by hash, checking address balances, inspecting blocks, and resolving ERC-20 token holdings. Ideal for crypto analysts, developers, and researchers requiring programmatic, detailed insights into on-chain financial activity and protocol interactions.

This tool detects profitable arbitrage opportunities across centralized exchanges (CEX), decentralized exchanges (DEX), and cross-chain markets in real-time. It aggregates price data, calculates net profit after fees, and identifies various arbitrage paths, including direct, triangular, and cross-chain opportunities. It is designed for quantitative traders needing to monitor market efficiency and exploit price discrepancies.