Skill UI

Guides data engineering patterns for ETL pipelines, Spark analytics, and quality checks, showcasing Python and SQL snippets to extract, transform, load, validate, and model sales data in modern warehouses.

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

A comprehensive skill set for mastering enterprise-grade Scala development. Expertise covers functional programming, distributed systems, and big data processing using technologies like Pekko, Akka, Spark, ZIO, and Cats Effect. Ideal for designing and implementing resilient, scalable microservices based on Domain-Driven Design principles.

Automates Sendspark workflows through Composio's Rube MCP toolkit by searching tools first, ensuring an active Sendspark connection, and issuing RUBE_MULTI_EXECUTE_TOOL calls with schema-compliant arguments.

This skill provides expertise in building production-grade, scalable data systems and pipelines. It covers the full spectrum of data engineering, including designing robust batch ETL/ELT workflows, implementing real-time streaming architectures (Kafka/Flink), and utilizing modern tools like Airflow, Spark, and dbt. Essential for data architecture design, data lakehouse implementation, and data governance.

Guides building, tuning, and validating production Apache Spark jobs, covering DataFrame/RDD choices, partitioning, broadcast joins, skew handling, caching, and cluster configuration for big data pipelines.

Comprehensive guide for setting up Continuous Integration/Continuous Deployment (CI/CD) pipelines for Palantir Foundry integrations using GitHub Actions. This workflow covers running PySpark unit tests, performing SDK integration tests with mocked APIs, and implementing custom linting rules to enforce Foundry-specific best practices. Use this when ensuring code quality and validating complex business logic changes in an automated build process.

This guide provides a comprehensive workflow for building robust ETL data pipelines within Palantir Foundry. It teaches how to utilize Python transforms, including the @transform_df decorator, for dataset-to-dataset processing. Key topics covered include multi-input joins, implementing incremental updates for efficiency, low-level file I/O, and configuring Spark resources for heavy computations. Essential for data engineers working with complex data modeling in Foundry.

This guide provides advanced techniques for optimizing performance within Palantir Foundry. It covers efficient pagination, implementing client-side caching, using batch object retrieval (solving the N+1 problem), and tuning Spark transforms with memory configuration and incremental updates. Use this when experiencing slow API responses or inefficient data pipeline builds in Foundry.