Skill UI

A comprehensive guide to building production-ready Apache Airflow Directed Acyclic Graphs (DAGs). This skill covers best practices for operators, sensors, testing, deployment, and designing robust data pipelines. Use this when orchestrating complex workflows, managing ETL processes, or ensuring reliable batch job execution in a production environment.

Expert in designing, building, and optimizing scalable data pipelines, modern data warehouses, and real-time streaming architectures. Mastery covers the complete modern data stack, including Spark, dbt, Airflow, and cloud services (AWS, Azure, GCP). Focuses on ELT/ETL, data lakehouse implementation (Delta Lake, Iceberg), data governance, and ensuring high data quality and operational reliability.

Expert guidance on designing highly scalable, distributed NoSQL databases, specifically covering Apache Cassandra and Amazon DynamoDB. This skill emphasizes the 'Query-First' modeling approach, teaching best practices like partitioning, denormalization, single-table design, and avoiding common anti-patterns (e.g., hot partitions, joins) to ensure optimal read/write performance at massive scale.

Guides building, tuning, and validating production Apache Spark jobs, covering DataFrame/RDD choices, partitioning, broadcast joins, skew handling, caching, and cluster configuration for big data pipelines.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

Polars is a blazing fast in-memory DataFrame library based on Apache Arrow that replaces pandas for 1-100GB workloads, offering expression-based lazy execution, parallel operators, and rich I/O for ETL pipelines.

This tool provides a comprehensive framework for analyzing web server logs (Apache/Nginx) to detect various security threats. It uses advanced regex matching against OWASP attack signatures to identify SQL injection, Local File Inclusion, and Directory Traversal attempts, alongside behavioral analysis for brute-force attacks and web scanner fingerprints. The system also enriches data using GeoIP for source attribution, generating a prioritized report for SOC analysts.

TLS 1.3 is the latest Transport Layer Security protocol, offering significant security and performance improvements over TLS 1.2. This guide details how to configure and validate TLS 1.3 on various environments, including Nginx, Apache, and Python applications. It covers advanced topics such as Perfect Forward Secrecy (PFS), HSTS implementation, modern cipher suites, and best practices for securing modern web services.

Polars is a lightning-fast DataFrame library built on Apache Arrow for Python. It provides an expression-based API, lazy query optimization, and parallel computation capabilities for efficient data manipulation, analytics, and ETL pipelines. It is designed to handle large, memory-intensive datasets, making it an ideal alternative for migrating from pandas or optimizing complex data workflows.

A comprehensive guide to Test-Driven Development (TDD) for Quarkus 3.x services, focusing on achieving high coverage (80%+). It covers unit testing with Mockito, integration testing of REST endpoints, and advanced testing of event-driven architectures using Apache Camel and RabbitMQ.

This powerful tool compares identical tables across two SQL Server instances (e.g., Production vs. Staging). It efficiently detects missing rows, extra columns, schema drifts, and data mismatches. Leveraging Python and Apache Arrow for fast columnar data transfer, it provides a comprehensive reconciliation report, making it essential for data migration validation and ETL verification.