Skill UI

An expert guide for performing complex data analysis, cleaning, and transformation using pandas DataFrames. This skill covers advanced techniques such as vectorized operations, efficient merging, time series resampling, group-by aggregations, and memory optimization, ensuring production-grade code and robust data validation.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

Polars is a lightning-fast DataFrame library designed for efficient, memory-efficient data processing in Python and Rust. Built upon Apache Arrow, it supports lazy evaluation and parallel execution, making it ideal for ETL pipelines, large-scale data transformations, and scenarios where standard Pandas performance is insufficient. It excels at handling datasets up to 100GB that fit within available RAM.