Skill UI

This skill provides a comprehensive guide on implementing robust data governance patterns within Delta Lake, leveraging Unity Catalog. It covers essential compliance topics such as data classification tagging, handling GDPR 'right-to-deletion' requests for PII, and establishing automated data retention policies. Use this when designing secure, compliant, and auditable data pipelines in a data lake environment.

A comprehensive guide for establishing robust, enterprise-grade access control within Databricks. Covers implementing Single Sign-On (SSO) via SCIM, leveraging Unity Catalog for fine-grained Role-Based Access Control (RBAC), defining cluster policies, and enforcing data governance measures like Row-Level Security (RLS) and column masking. Essential for large-scale, secure data platforms.

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

This guide outlines a production-ready lakehouse architecture using Unity Catalog, Delta Lake, and the Medallion pattern. It provides best practices for organizing Databricks workspaces, implementing three-level governance, defining compute strategies, and adopting CI/CD best practices with Asset Bundles for robust team collaboration.

A comprehensive guide to implementing best practices for security within Databricks. Learn how to manage secrets using scoped storage, enforce the principle of least privilege via Unity Catalog grants, audit API tokens, and apply advanced data protection techniques like column-level masking and row-level security for enhanced data governance and compliance.

This comprehensive guide provides structured steps and code examples for upgrading Databricks Runtime versions and performing critical data migrations. It covers best practices for version compatibility checking, transitioning from Hive Metastore to Unity Catalog using methods like SYNC and CTAS, and updating deprecated API endpoints, ensuring a smooth platform evolution for data pipelines.

This comprehensive guide sets up a robust CI/CD pipeline for Deepgram integrations using GitHub Actions. It covers unit tests (mocked SDK), full integration tests against the live Deepgram API (including transcription, diarization, and TTS), and smoke tests. Use this when establishing automated testing, ensuring deployment stability, or implementing continuous delivery workflows for Deepgram-powered applications.

This guide establishes a robust local development environment for Deepgram integrations. It covers setting up test fixtures using sample audio, mocking API responses for isolated unit testing, configuring watch-mode development servers, and running comprehensive unit and integration tests using Vitest. Use this setup when initiating a Deepgram feature or requiring a reliable, fast development cycle.

A comprehensive guide for mastering major dependency version upgrades in software development. This skill covers compatibility analysis, implementing staged upgrade strategies, resolving dependency conflicts, and setting up robust testing matrices (Unit, Integration, E2E) to ensure a smooth and reliable modernization process for any codebase.

Blueprint for designing comprehensive test suites, guiding when to add unit, integration, or E2E tests, selecting frameworks, managing data/mocks, and validating coverage and reliability.

This comprehensive guide details how to configure robust CI/CD pipelines for Documenso integrations using GitHub Actions. It covers best practices for automated unit testing with mocks, comprehensive integration testing against a staging environment, and secure deployment workflows, ensuring code quality and reliability from commit to production.