Skill UI

Automates guidance for data catalog updater tasks, covering ETL, transformation, orchestration, and streaming workflows to keep pipeline metadata accurate.

Expert in designing, building, and optimizing scalable data pipelines, modern data warehouses, and real-time streaming architectures. Mastery covers the complete modern data stack, including Spark, dbt, Airflow, and cloud services (AWS, Azure, GCP). Focuses on ELT/ETL, data lakehouse implementation (Delta Lake, Iceberg), data governance, and ensuring high data quality and operational reliability.

An expert in designing, implementing, and optimizing scalable, reliable, and cost-effective data pipelines for both batch and streaming data. Covers advanced topics including ETL/ELT patterns, Kappa/Lambda architectures, workflow orchestration (Airflow/Prefect), transformation using dbt/Spark, and managing modern storage formats like Delta Lake and Iceberg. Focuses heavily on data quality, monitoring, and cost optimization strategies.

Offers automated guidance for implementing data lineage trackers across ETL and streaming pipelines, including best practices on transformations, orchestration, and validation for production environments.

Guides engineers through data partitioner setups within ETL workflows, offering best practice advice, ready-to-run code, and validation for streaming and batch pipelines.

Automated Data Quality Checker offers guidance, best practices, and production-ready code for validating pipelines whenever you work with ETL, streaming, or data transformation scenarios.

This comprehensive guide outlines the end-to-end process of building production-grade ETL data pipelines using Databricks and Delta Lake. It implements the robust Medallion Architecture (Bronze, Silver, Gold) for scalable data management. Techniques covered include Auto Loader for incremental streaming ingestion, MERGE INTO for upsert semantics, data cleansing, and advanced aggregation into analytics-ready layers. Ideal for establishing reliable data pipelines in a data lakehouse environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

Guides you through generating dbt models inside data pipelines, covering setup, code generation, configuration, and validation while following best practices for ETL, orchestration, and streaming workloads.

Automates dbt test creation within data pipelines, offering guided best practices, code generation, and validation to keep ETL, transformation, and streaming workflows reliable.

Uses the Gemini API to autonomously plan, search, read, and synthesize detailed research reports with streaming status updates, structured outputs, and the ability to continue previous investigations for market analysis, technical reviews, or due diligence.

This guide demonstrates how to implement robust, real-time, low-latency streaming transcription using Deepgram's WebSocket API. It covers advanced audio handling, including microphone input capture (via Sox), speaker diarization, utterance boundary detection, and implementing automatic reconnection logic. Essential for building professional voice AI applications, live captioning services, or voice interfaces.