Skill UI

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

FAISS (Facebook AI Similarity Search) is a library designed for efficient similarity search and clustering of dense vectors. It supports indexing and searching on billions of vectors at scale, featuring GPU acceleration and multiple index types (Flat, IVF, HNSW, PQ) to balance speed and accuracy. It is essential for building high-performance, large-scale applications requiring fast k-NN retrieval or pure vector similarity matching, especially in fields like RAG and recommendation systems.

Automates clustering workflows on supplied datasets with K-means, DBSCAN, or hierarchical algorithms, handling preprocessing, validation, metrics, and visualizations to expose groupings and anomalies.

Sentence Transformers provides 5000+ pre-trained models that generate sentence, text, and image embeddings for RAG, semantic search, clustering, and multilingual production workloads without relying on external APIs.

Aeon is a scikit-learn compatible toolkit for time series machine learning, covering classification, regression, clustering, forecasting, anomaly detection, segmentation, similarity search, and feature extraction on temporal data.

Datamol wraps RDKit with Pythonic APIs tailored for drug discovery, offering SMILES/SELFIES/InChI parsing, standardization, descriptor and fingerprint calculation, clustering, 3D conformer workflows, and parallel/cloud-aware batch I/O while returning native rdkit.Chem.Mol objects.

Geniml builds machine learning workflows around BED and scATAC datasets to learn embeddings for regions, metadata, and cells, enabling similarity searches, clustering, consensus peaks, and downstream analysis of genomic intervals.

Scanpy is a scalable Python toolkit for single-cell RNA-seq analyses, covering QC, normalization, dimensionality reduction, clustering, marker discovery, annotation, and publication-ready visualization across h5ad/10X/CSV datasets.

Use UMAP for fast nonlinear manifold reduction to produce embeddings that keep local and global structure, making it ideal for visualization, clustering preprocessing, supervised/parametric workflows, or high-dimensional feature engineering.

Batch-categorize uncategorized transactions by clustering similar descriptions, researching unknown merchants, and creating rules or annotations so future payments auto-classify with minimal manual review.

A comprehensive skill for optimizing Role-Based Access Control (RBAC) structures. It applies advanced data mining techniques, including clustering (k-means, agglomerative) and Formal Concept Analysis (FCA), to analyze existing user-permission assignments. The goal is to discover the minimal set of necessary roles, thereby preventing 'role explosion,' enforcing the least privilege principle, and ensuring robust identity governance for secure systems.

This skill helps refine vague, raw ideas into sharp, actionable concepts. It guides users through structured divergent thinking (expanding options, generating variations) and convergent thinking (stress-testing assumptions, clustering directions). The process culminates in a detailed one-pager including Problem Statements, MVP Scope, and key assumptions to validate, ensuring focus and clarity before development.