Skill UI

Automates the creation of Airflow DAGs for ETL and streaming data workflows, offering best-practice guidance, production-ready code samples, and validation against common pipeline standards.

A comprehensive guide to building production-ready Apache Airflow Directed Acyclic Graphs (DAGs). This skill covers best practices for operators, sensors, testing, deployment, and designing robust data pipelines. Use this when orchestrating complex workflows, managing ETL processes, or ensuring reliable batch job execution in a production environment.

Guides you through building Airflow operators for data pipelines with automated best practices, production-ready code/config generation, and validation for ETL, transformation, orchestration, and streaming scenarios.

This comprehensive guide identifies and explains common anti-patterns, gotchas, and integration mistakes when using Clay's data enrichment platform. It details the root causes of credit waste, workflow failures, and data loss, providing actionable fixes for auditing configurations, onboarding new team members, or optimizing existing pipelines.

A comprehensive system for implementing policy guardrails in data enrichment pipelines. It enforces critical controls including credit spending limits, blocks the enrichment of sensitive PII (e.g., SSN, bank details), and validates input data quality. Essential for maintaining compliance (GDPR/CCPA) and managing costs when processing large volumes of personal and business data.

Guides ML training teams through data augmentation pipelines, offering best practices, code/config generation, validation, and experiment-tracking support so pipelines stay production-ready.

This skill details how to set up automated CI/CD pipelines for Databricks data pipelines. It leverages Declarative Automation Bundles (DABs) and GitHub Actions to manage the entire deployment lifecycle. Key functionalities include running unit tests on pull requests, validating and deploying to staging environments, and promoting validated code to production upon merge, ensuring reliable and automated data infrastructure.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

A comprehensive workflow for executing advanced, semantic web searches using Exa. This tool moves beyond simple keyword matching, enabling deep context retrieval by understanding the intent behind the query. Key features include RAG context retrieval, content extraction (summaries, highlights, full text), and robust filtering by date range, specific domains, and content categories. Ideal for building knowledge base search features and complex research pipelines.

A comprehensive pipeline for processing, validating, and optimizing scraped web content from Firecrawl. This tool handles markdown cleaning, structured data extraction using Zod validation, content deduplication, and chunking tailored for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems. It ensures crawled data is standardized, clean, and ready for downstream consumption or knowledge base indexing.

This comprehensive reference architecture details a multi-stage pipeline for web scraping and content ingestion using Firecrawl. It seamlessly integrates single-page scraping, site-wide crawling, URL mapping, and structured data extraction. The subsequent processing layer handles cleaning, deduplication, and chunking, making this blueprint ideal for designing robust knowledge bases, RAG systems, and complex AI data pipelines.