Skill UI

Use Declarative Automation Bundles (DABs) to manage and deploy Databricks assets, including jobs, DLT pipelines, and ML models, using an Infrastructure-as-Code approach. This skill allows defining resources and deployment targets (dev, staging, prod) in a single YAML file, enabling robust CI/CD practices and controlled promotion of changes across environments.

A comprehensive guide to setting up a minimal working example on Databricks. This skill covers creating a development cluster, uploading and executing notebooks (Python SDK), running tasks as jobs, and setting up a Serverless SQL Warehouse. It is ideal for new users or initializing a basic data engineering project.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

Master the implementation of Firecrawl webhooks for robust, real-time handling of asynchronous crawling and batch scraping jobs. Instead of inefficiently polling the crawl status, configure webhook endpoints to receive instant POST events (e.g., `crawl.page`, `crawl.completed`). This guide covers event types, signature verification using HMAC-SHA256, and processing streamed data for building advanced data pipelines.

Automates spark job creation within data pipelines by guiding etl, transformation, orchestration, and streaming workflows; includes industry best practices, code templates, and validation to deliver production-ready jobs.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

This advanced workflow orchestrates complex, multi-node GPU clusters on Vast.ai. It is designed for large-scale distributed machine learning training, automatically handling spot instance preemption and resuming jobs from checkpoints. Furthermore, it provides comprehensive cost analysis by tracking billing history, allowing users to optimize GPU spending and manage cluster teardown efficiently.

Arboreto infers gene regulatory networks from bulk or single-cell RNA-seq data using scalable GRNBoost2 and GENIE3 algorithms, letting researchers filter transcription-factor targets, run distributed Dask jobs, and export ranked links for downstream analysis.

Provides a detailed, multi-platform workflow to monitor long-running computational experiments. It checks the status of jobs across various environments (SSH, vast.ai, Modal), collects screen logs, fetches structured JSON results, and pulls detailed training metrics (loss curves, accuracy) from Weights & Biases (W&B). Essential for MLOps and automated result review.

This comprehensive tool manages the complete lifecycle of ML experiments, providing robust deployment capabilities across diverse environments. It supports running training jobs on local GPUs, dedicated remote servers, Vast.ai, and serverless platforms like Modal. The workflow handles environment detection, code synchronization (rsync/git), dependency pinning, and integrates advanced logging tools (e.g., W&B), ensuring reliable, scalable, and reproducible ML training runs.

This expert guides users through comprehensive customer research, whether analyzing existing assets (transcripts, surveys, tickets) or gathering new intel from online sources (Reddit, G2, forums). It helps uncover deep customer pains, desired outcomes, and underlying needs (Jobs to Be Done) to ground product strategy, marketing copy, and positioning in real-world evidence.