Skill UI

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

A comprehensive guide and implementation pattern for managing rate limits, throttling, and ensuring idempotency when interacting with the Databricks API. This skill demonstrates techniques like exponential backoff with jitter, token-bucket rate limiting, and using idempotency tokens to make bulk and concurrent API calls reliable and efficient, minimizing HTTP 429 errors.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

This guide provides essential best practices for building reliable, fault-tolerant applications in Go using DBOS durable workflows. It covers architectural patterns for integrating DBOS into existing Go code, defining complex workflows, managing steps, and utilizing queues for robust concurrency control. Essential reading for developers setting up, configuring, or testing mission-critical asynchronous processes.

This skill guides the definition of relationships and integration contracts between distinct bounded contexts within a complex system using Domain-Driven Design (DDD) patterns. It is crucial for preventing domain leakage, planning anti-corruption layers (ACLs), and clarifying ownership boundaries when multiple services interact, ensuring a robust and scalable system architecture.

This skill guides the application of Domain-Driven Design (DDD) tactical patterns, including defining aggregates, modeling value objects, managing repositories, and utilizing domain events. It helps developers translate complex domain rules into clean, robust, and behavior-rich code structures, ensuring invariants are explicitly enforced.

Uses the Gemini API to autonomously plan, search, read, and synthesize detailed research reports with streaming status updates, structured outputs, and the ability to continue previous investigations for market analysis, technical reviews, or due diligence.

A comprehensive workflow for production-grade, pre-recorded Speech-to-Text (STT) transcription using the Deepgram API. This skill supports transcribing audio from URLs or local files, offering advanced features such as speaker diarization, automatic summaries, sentiment analysis, and topic detection. It also includes robust batch processing methods for handling large datasets.

This skill provides comprehensive methods to optimize Deepgram API usage and manage costs. It implements budget-aware transcription pipelines that preemptively check projected spending against set limits, utilizes audio preprocessing techniques (like silence removal via ffmpeg) to minimize billable minutes, and includes API calls to monitor current usage and spending against defined budgets. Ideal for cost-sensitive, production-grade deployments.

Provides comprehensive instructions for deploying Deepgram transcription services into various production environments. This guide covers building production-ready Docker images, configuring Docker Compose, setting up robust Kubernetes manifests (including HPA and secrets), and implementing serverless handlers for platforms like AWS Lambda. Essential for scaling Deepgram services reliably and securely in the cloud.