Skill UI

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive diagnostic bundle collects critical evidence from the Databricks environment, including environment setup, cluster state, job run history, and Delta table diagnostics. It generates a redacted, safe tar.gz file specifically designed for submitting detailed support tickets and efficiently troubleshooting complex data pipeline issues.

A comprehensive guide to setting up a minimal working example on Databricks. This skill covers creating a development cluster, uploading and executing notebooks (Python SDK), running tasks as jobs, and setting up a Serverless SQL Warehouse. It is ideal for new users or initializing a basic data engineering project.

A comprehensive runbook designed for on-call engineers to manage Databricks incidents, outages, and major job failures. It provides structured procedures for immediate triage, decision tree guidance (covering cluster failures, code errors, and data quality issues), and steps for evidence collection and postmortem documentation, ensuring rapid and systematic incident resolution.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

This comprehensive checklist guides the deployment of Databricks data pipelines and jobs to a production environment. It covers critical stages including pre-deployment security hardening (Service Principals, Unity Catalog), validating infrastructure (cluster types, autoscaling), ensuring code quality (unit/integration tests, Delta Lake best practices), executing controlled deployment, and performing post-deployment health checks and monitoring setup. Ideal for CI/CD pipelines and go-live procedures.

Ray Data offers scalable distributed data processing for ML workloads, streaming across CPU/GPU clusters and integrating with PyTorch, TensorFlow, and Ray Train. Use it for batch inference, ETL, and multi-modal preprocessing from a laptop up to hundreds of nodes.

Ray Train orchestrates distributed PyTorch, TensorFlow, and HuggingFace workloads from laptops to clusters, automating GPU placement, fault tolerance, checkpointing, and hyperparameter sweeps so teams can scale massive model training with minimal code changes.

Automates clustering workflows on supplied datasets with K-means, DBSCAN, or hierarchical algorithms, handling preprocessing, validation, metrics, and visualizations to expose groupings and anomalies.

This advanced workflow orchestrates complex, multi-node GPU clusters on Vast.ai. It is designed for large-scale distributed machine learning training, automatically handling spot instance preemption and resuming jobs from checkpoints. Furthermore, it provides comprehensive cost analysis by tracking billing history, allowing users to optimize GPU spending and manage cluster teardown efficiently.

Aeon is a scikit-learn compatible toolkit for time series machine learning, covering classification, regression, clustering, forecasting, anomaly detection, segmentation, similarity search, and feature extraction on temporal data.

Datamol wraps RDKit with Pythonic APIs tailored for drug discovery, offering SMILES/SELFIES/InChI parsing, standardization, descriptor and fingerprint calculation, clustering, 3D conformer workflows, and parallel/cloud-aware batch I/O while returning native rdkit.Chem.Mol objects.