Skill UI

Guides data engineering patterns for ETL pipelines, Spark analytics, and quality checks, showcasing Python and SQL snippets to extract, transform, load, validate, and model sales data in modern warehouses.

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

A comprehensive skill set for mastering enterprise-grade Scala development. Expertise covers functional programming, distributed systems, and big data processing using technologies like Pekko, Akka, Spark, ZIO, and Cats Effect. Ideal for designing and implementing resilient, scalable microservices based on Domain-Driven Design principles.

Automates Sendspark workflows through Composio's Rube MCP toolkit by searching tools first, ensuring an active Sendspark connection, and issuing RUBE_MULTI_EXECUTE_TOOL calls with schema-compliant arguments.

This comprehensive guide details the expertise needed for a Senior Data Engineer. It covers building, optimizing, and governing scalable data pipelines and robust ETL/ELT systems using modern tools like Spark, Kafka, Airflow, and dbt. Includes decision frameworks for architecting data solutions (Batch vs Streaming, Lakehouse vs Warehouse) and implementing advanced data modeling patterns.

Guides building, tuning, and validating production Apache Spark jobs, covering DataFrame/RDD choices, partitioning, broadcast joins, skew handling, caching, and cluster configuration for big data pipelines.

Comprehensive guide for setting up Continuous Integration/Continuous Deployment (CI/CD) pipelines for Palantir Foundry integrations using GitHub Actions. This workflow covers running PySpark unit tests, performing SDK integration tests with mocked APIs, and implementing custom linting rules to enforce Foundry-specific best practices. Use this when ensuring code quality and validating complex business logic changes in an automated build process.

This guide provides a comprehensive workflow for building robust ETL data pipelines within Palantir Foundry. It teaches how to utilize Python transforms, including the @transform_df decorator, for dataset-to-dataset processing. Key topics covered include multi-input joins, implementing incremental updates for efficiency, low-level file I/O, and configuring Spark resources for heavy computations. Essential for data engineers working with complex data modeling in Foundry.

This guide provides advanced techniques for optimizing performance within Palantir Foundry. It covers efficient pagination, implementing client-side caching, using batch object retrieval (solving the N+1 problem), and tuning Spark transforms with memory configuration and incremental updates. Use this when experiencing slow API responses or inefficient data pipeline builds in Foundry.

A comprehensive tool for the full lifecycle management of applications within the Lark ecosystem (Feishu/Miaoda). It supports three development paths: local full-stack development, HTML static hosting, and cloud-based AI session generation. Use this when creating, updating, deploying, or publishing any complex system, tool, or platform application.

A comprehensive guide for generating dynamic, inline SVG graphics (such as sparklines, gauges, and charts) within Power BI reports using DAX measures. This technique allows developers to embed complex visualizations directly into standard visuals (Table, Card, Matrix) by having the DAX function return an SVG markup string. Best practices emphasize prioritizing existing User Defined Function (UDF) libraries (e.g., IBCS, DaxLib.SVG) over writing custom DAX code.