Skill UI

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

Automated assistance for pyspark transformer work within data pipelines, activating when you request transformer patterns or best practices and covering ETL, data transformation, orchestration, and streaming scenarios.

A structured guide for setting up a robust local development environment for Palantir Foundry data pipelines. This workflow enables users to run Python transforms and unit tests using PySpark against sample data, and mock the Foundry API client, ensuring fast, isolated iteration cycles before deploying code to the official Foundry environment. Ideal for data engineers and developers building complex ETL logic.

Execute arbitrary PySpark or pure Python code directly on Fabric Spark compute using the Livy API. This method creates ephemeral sessions without generating persistent notebook artifacts. It provides full read/write access to lakehouse Delta tables, making it ideal for agent-driven ETL pipelines, quick data transformations, or analyses where session cleanup is critical. Users must manage the session lifecycle (create, run, and delete).