Skill UI

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

This skill provides a comprehensive guide to configuring event-driven workflows and robust notification systems for Databricks jobs. It covers setting up various notification destinations (Slack, Teams, Email, PagerDuty) and triggering alerts based on job lifecycle events (success, failure, start). Furthermore, it demonstrates advanced monitoring using system audit tables and creating automated SQL alerts for critical data pipeline monitoring.

Guides you through generating dbt models inside data pipelines, covering setup, code generation, configuration, and validation while following best practices for ETL, orchestration, and streaming workloads.

Automates dbt test creation within data pipelines, offering guided best practices, code generation, and validation to keep ETL, transformation, and streaming workflows reliable.

A guide to production-ready patterns for dbt (data build tool). It covers essential practices for robust data pipeline development, including model organization (staging, intermediate, marts), implementing advanced testing strategies, ensuring data quality, and optimizing incremental processing for large datasets.

This comprehensive guide sets up a robust CI/CD pipeline for Deepgram integrations using GitHub Actions. It covers unit tests (mocked SDK), full integration tests against the live Deepgram API (including transcription, diarization, and TTS), and smoke tests. Use this when establishing automated testing, ensuring deployment stability, or implementing continuous delivery workflows for Deepgram-powered applications.

This skill provides comprehensive methods to optimize Deepgram API usage and manage costs. It implements budget-aware transcription pipelines that preemptively check projected spending against set limits, utilizes audio preprocessing techniques (like silence removal via ffmpeg) to minimize billable minutes, and includes API calls to monitor current usage and spending against defined budgets. Ideal for cost-sensitive, production-grade deployments.

Implements best practices for secure audio transcription using Deepgram. This guide covers PII redaction (built-in and custom patterns), secure audio upload to S3 with encryption, managing data retention policies, and ensuring compliance with major regulations like GDPR and HIPAA. It provides a comprehensive pipeline for handling sensitive audio data in enterprise applications.

Provides reference architectures for building high-scale, reliable transcription systems using Deepgram. It covers four main patterns: synchronous REST for short files, async queues (BullMQ) for batch processing, WebSocket for real-time streaming, and a hybrid router for mixed workloads. Use this when designing production-grade audio processing pipelines.

Provides production-ready coding patterns for integrating Deepgram using TypeScript and Python. This guide covers essential best practices, including implementing singleton client management, performing text-to-speech synthesis with Aura, and setting up advanced audio intelligence pipelines (e.g., summarization, topic detection, sentiment analysis). Use this when establishing robust Deepgram integration standards, refactoring existing SDK usage, or migrating to newer SDK versions.

Guides engineers through DeepSpeed distributed training and NVMe I/O optimizations, covering ZeRO, pipeline parallelism, mixed precision, and DeepNVMe handles for synchronous and asynchronous tensor storage.

Expert in modern CI/CD pipelines, GitOps workflows, and platform engineering. Specializes in designing and implementing secure, zero-downtime deployment strategies for enterprise-scale applications. Proficient with Kubernetes, ArgoCD, Terraform, and advanced security compliance practices.