Skill UI

An expert guide for performing complex data analysis, cleaning, and transformation using pandas DataFrames. This skill covers advanced techniques such as vectorized operations, efficient merging, time series resampling, group-by aggregations, and memory optimization, ensuring production-grade code and robust data validation.

Guides building, tuning, and validating production Apache Spark jobs, covering DataFrame/RDD choices, partitioning, broadcast joins, skew handling, caching, and cluster configuration for big data pipelines.

Dask is a Python library that scales pandas/NumPy workflows beyond RAM by parallelizing DataFrames, Arrays, Bags, and custom tasks across cores or clusters for ETL, log analysis, and distributed ML.

FlowIO reads and writes Flow Cytometry Standard (FCS) files, exposing metadata, channels, and NumPy event arrays while allowing CSV/DataFrame conversion, making it ideal for cytometry pipelines and backend preprocessing services.

Polars is a blazing fast in-memory DataFrame library based on Apache Arrow that replaces pandas for 1-100GB workloads, offering expression-based lazy execution, parallel operators, and rich I/O for ETL pipelines.

Vaex is a high-performance Python library for lazy out-of-core DataFrames that process billions of rows without RAM. Use it to analyze massive CSV/HDF5/Arrow/Parquet tables, compute fast statistics, visualize big data, and build scikit-learn pipelines.

This script performs statistical analysis on Zeek connection logs (conn.log) to detect Command and Control (C2) beaconing patterns. It leverages the ZAT library to load data into Pandas DataFrames, calculates inter-arrival time statistics, and flags connections with low standard deviation relative to the mean. This technique is crucial for network threat hunting and security incident investigation when identifying malicious, periodic callbacks.

Dask is a powerful Python library designed for scalable, parallel, and distributed computing. It enables workflows for datasets that exceed available RAM or require processing across multiple machines. Users can scale familiar pandas and NumPy operations (DataFrames, Arrays) and process unstructured data (Bags) efficiently, making it ideal for large-scale ETL, analytics, and distributed machine learning tasks.

Polars is a lightning-fast DataFrame library built for Python and Rust, leveraging Apache Arrow. It provides an expression-based API and lazy evaluation framework for highly efficient data manipulation. It is ideal for complex data pipelines, large-scale ETL tasks, and migrating from Pandas due to its superior performance, parallel execution capabilities, and optimized query planning.

polars-bio is a high-performance Python library built on Polars for genomic interval operations and bioinformatics file I/O. It provides a familiar DataFrame-centric API for interval arithmetic (overlap, nearest, merge, complement, etc.) and supports reading/writing major bioinformatics formats like BED, VCF, BAM, and GFF. Key features include streaming, cloud-native I/O, and SQL query support, making it a fast alternative for large-scale genomic data processing.

Seaborn is a high-level Python visualization library built on Matplotlib, designed specifically for creating publication-quality statistical graphics. It excels at dataset-oriented plotting, enabling users to perform complex multivariate analyses, visualize data distributions (like box plots, violin plots), and explore relationships with minimal code. It simplifies the process of generating multi-panel figures and statistical plots directly from pandas DataFrames.

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames. It enables processing and visualization of massive tabular datasets (billions of rows) that exceed available RAM. Users can perform fast statistics, aggregations, and build robust ML pipelines on big data files (CSV, Parquet, HDF5) without relying on distributed clusters, making it ideal for single-machine, large-scale analytics.