Skill UI

This comprehensive Python SDK serves as a client library for managing the full lifecycle of Machine Learning resources on Azure Machine Learning. It allows users to programmatically handle workspaces, register data and models, provision compute clusters, define complex training jobs, and build reproducible, end-to-end MLOps pipelines.

This comprehensive Python SDK serves as a client library for managing the full lifecycle of Machine Learning resources on Azure Machine Learning. It allows users to programmatically handle workspaces, register data and models, provision compute clusters, define complex training jobs, and build reproducible, end-to-end MLOps pipelines.

Creates production-ready GitOps workflows for ArgoCD or Flux, producing manifests, repository layout guidance, and deployment instructions tailored to the user’s infrastructure so Kubernetes clusters stay in sync with Git.

This advanced system transforms session interactions into reusable, atomic knowledge units called "Instincts." By hooking into tool usage and session activity, it observes patterns (like preferred coding styles, error fixes, or repeated workflows). These instincts are then clustered and "evolved" into structured Skills, Commands, or specialized Agents, capturing deep contextual knowledge with weighted confidence scoring to enhance development productivity.

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This guide provides a comprehensive framework for minimizing cloud expenditure on Databricks. It covers identifying cost drivers via system billing tables, implementing cluster policies, utilizing spot instances for batch processing, right-sizing SQL warehouses, and automating the shutdown of idle development clusters. Use it when managing cloud spending or optimizing data platform usage.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

A comprehensive guide for establishing robust, enterprise-grade access control within Databricks. Covers implementing Single Sign-On (SSO) via SCIM, leveraging Unity Catalog for fine-grained Role-Based Access Control (RBAC), defining cluster policies, and enforcing data governance measures like Row-Level Security (RLS) and column masking. Essential for large-scale, secure data platforms.

A comprehensive guide to setting up a minimal working example on Databricks. This skill covers creating a development cluster, uploading and executing notebooks (Python SDK), running tasks as jobs, and setting up a Serverless SQL Warehouse. It is ideal for new users or initializing a basic data engineering project.

A comprehensive runbook designed for on-call engineers to manage Databricks incidents, outages, and major job failures. It provides structured procedures for immediate triage, decision tree guidance (covering cluster failures, code errors, and data quality issues), and steps for evidence collection and postmortem documentation, ensuring rapid and systematic incident resolution.

This skill provides comprehensive observability for the Databricks platform. It enables users to monitor job run histories, track cluster utilization and associated costs (DBU), detect slow SQL queries, and audit access logs. Use it for setting up robust monitoring dashboards, ensuring pipeline health, and optimizing resource consumption across the entire data stack.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.