Skill UI

Expert in designing, building, and optimizing scalable data pipelines, modern data warehouses, and real-time streaming architectures. Mastery covers the complete modern data stack, including Spark, dbt, Airflow, and cloud services (AWS, Azure, GCP). Focuses on ELT/ETL, data lakehouse implementation (Delta Lake, Iceberg), data governance, and ensuring high data quality and operational reliability.

Guides building, tuning, and validating production Apache Spark jobs, covering DataFrame/RDD choices, partitioning, broadcast joins, skew handling, caching, and cluster configuration for big data pipelines.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

Expert data engineer specializing in scalable data pipelines, modern data warehouses, lakehouse architectures, and real-time streaming. Proficient in Apache Spark, dbt, Airflow, Kafka, and cloud-native platforms (AWS, Azure, GCP). Focuses on reliable, performant, and cost-effective data solutions with strong emphasis on data quality and governance.

Master enterprise-grade Scala development with functional programming, Cats Effect and ZIO effect systems, distributed systems via Apache Pekko, Akka and Spark, plus reactive, event-driven and microservice architectures.

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.