Skill UI

Automated assistance for pyspark transformer work within data pipelines, activating when you request transformer patterns or best practices and covering ETL, data transformation, orchestration, and streaming scenarios.

Ray Data offers scalable distributed data processing for ML workloads, streaming across CPU/GPU clusters and integrating with PyTorch, TensorFlow, and Ray Train. Use it for batch inference, ETL, and multi-modal preprocessing from a laptop up to hundreds of nodes.

Automates Rocketlane operations through Composio's Rube MCP, guiding users to discover current schemas, verify toolkit connections, and execute multi-tool workflows while honoring schema, session, and pagination requirements.

Automates schema validation within data pipelines, providing guidance on best practices, workflow orchestration, and streaming ETL scenarios while producing production-ready code and configurations.

This comprehensive guide details the expertise needed for a Senior Data Engineer. It covers building, optimizing, and governing scalable data pipelines and robust ETL/ELT systems using modern tools like Spark, Kafka, Airflow, and dbt. Includes decision frameworks for architecting data solutions (Batch vs Streaming, Lakehouse vs Warehouse) and implementing advanced data modeling patterns.

Automates spark job creation within data pipelines by guiding etl, transformation, orchestration, and streaming workflows; includes industry best practices, code templates, and validation to deliver production-ready jobs.

Automates Spark SQL optimizer tasks in data pipelines by guiding best practices, producing production-ready code and configurations, and validating ETL, transformation, orchestration, and streaming workflows.

Automated guidance for SQL transform helper tasks within data pipelines, covering ETL, data transformation, workflow orchestration, and streaming, while following best practices, generating production-ready code, and validating results.

This comprehensive guide details the process of migrating applications to Vercel from various platforms, including Netlify, AWS, and Cloudflare. It covers critical steps such as configuration mapping (e.g., netlify.toml to vercel.json), migrating serverless functions, setting up environment variables, implementing DNS cutovers, and validating feature parity using advanced techniques like the Strangler Fig pattern, ensuring a smooth and reliable transition.

Integrates Windsurf's native deployment features and Cascade workflows to automate the entire build, test, and deployment cycle. Supports major platforms like Netlify, Vercel, AWS, and Google Cloud Run, enabling users to manage the full CI/CD pipeline, including preview deployments and rollbacks, directly within the IDE.

Dask is a Python library that scales pandas/NumPy workflows beyond RAM by parallelizing DataFrames, Arrays, Bags, and custom tasks across cores or clusters for ETL, log analysis, and distributed ML.

Polars is a blazing fast in-memory DataFrame library based on Apache Arrow that replaces pandas for 1-100GB workloads, offering expression-based lazy execution, parallel operators, and rich I/O for ETL pipelines.