Skill UI

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

An expert guide for performing complex data analysis, cleaning, and transformation using pandas DataFrames. This skill covers advanced techniques such as vectorized operations, efficient merging, time series resampling, group-by aggregations, and memory optimization, ensuring production-grade code and robust data validation.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

A comprehensive guide to optimizing deep learning workflows and maximizing efficiency on cloud GPU platforms like Vast.ai. Learn techniques for selecting optimal instances by performance-per-dollar, reducing startup latency, tuning data pipelines (DataLoader, pin_memory), and scaling multi-GPU training using PyTorch and Bash scripting.

This framework manages the ingestion, organization, synchronization, and search of knowledge across multiple storage layers, including local files, persistent memory, vector stores, and Git repositories. It provides a structured approach for ensuring knowledge durability, deduplication, and maintaining a 'single source of truth' across complex development and operational contexts.

Provides programmatic access to massive, versioned public single-cell and spatial transcriptomics datasets from CZ CELLxGENE. This skill allows users to efficiently query specific subsets of cell metadata, gene expression matrices, embeddings, and summary statistics across various organisms, tissues, and diseases. It is ideal for large-scale, memory-efficient biological research and model training without full dataset downloads.

This skill proactively detects and reports comprehensive system resources, including CPU cores, GPU types (NVIDIA, AMD, Apple Silicon), available memory, and disk space. It generates a detailed JSON report with strategic, context-aware recommendations for optimizing scientific computing workflows, such as recommending parallel processing libraries (Dask, joblib), out-of-core methods (Zarr), or appropriate GPU acceleration backends (Metal, ROCm). Use this before any large-scale data analysis, model training, or computationally intensive simulation to make informed architectural decisions.

Latch is a robust Python framework designed for building, deploying, and managing complex, scalable bioinformatics workflows as serverless pipelines. It supports integrating multiple analysis tools (like Nextflow/Snakemake) and handles cloud data management (LatchFile/LatchDir) and detailed resource configuration (GPU, CPU, memory). Ideal for running large-scale analyses such as RNA-seq, AlphaFold, and single-cell analysis in a reproducible, production-ready environment.

Polars is a lightning-fast DataFrame library built on Apache Arrow for Python. It provides an expression-based API, lazy query optimization, and parallel computation capabilities for efficient data manipulation, analytics, and ETL pipelines. It is designed to handle large, memory-intensive datasets, making it an ideal alternative for migrating from pandas or optimizing complex data workflows.

Vaex is a high-performance Python library designed for processing and visualizing enormous tabular datasets that exceed available RAM (gigabytes to terabytes). It excels at out-of-core DataFrame operations, lazy evaluation, fast statistical aggregation, and machine learning integration on massive datasets. Use it for analyzing large CSV, Parquet, or HDF5 files efficiently on a single machine, without memory constraints.

This guide provides a quick reference for diagnosing and fixing the top common errors encountered in ClickHouse, including syntax errors, memory limit issues, write permission failures, and data type mismatches. It offers specific error codes, root causes, and actionable SQL solutions for smooth database operation and complex analytics.