Skill UI

A comprehensive guide to setting up a fast, isolated local development loop for data engineering projects using Databricks Connect v2, Asset Bundles, and standard IDEs (like VS Code). This workflow allows developers to run PySpark code locally for debugging and unit testing, while managing resource deployment and ensuring consistency across development, staging, and production environments.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

Automated assistance for pyspark transformer work within data pipelines, activating when you request transformer patterns or best practices and covering ETL, data transformation, orchestration, and streaming scenarios.

Comprehensive guide for setting up Continuous Integration/Continuous Deployment (CI/CD) pipelines for Palantir Foundry integrations using GitHub Actions. This workflow covers running PySpark unit tests, performing SDK integration tests with mocked APIs, and implementing custom linting rules to enforce Foundry-specific best practices. Use this when ensuring code quality and validating complex business logic changes in an automated build process.

A structured guide for setting up a robust local development environment for Palantir Foundry data pipelines. This workflow enables users to run Python transforms and unit tests using PySpark against sample data, and mock the Foundry API client, ensuring fast, isolated iteration cycles before deploying code to the official Foundry environment. Ideal for data engineers and developers building complex ETL logic.

Execute arbitrary PySpark or pure Python code directly on Fabric Spark compute using the Livy API. This method creates ephemeral sessions without generating persistent notebook artifacts. It provides full read/write access to lakehouse Delta tables, making it ideal for agent-driven ETL pipelines, quick data transformations, or analyses where session cleanup is critical. Users must manage the session lifecycle (create, run, and delete).