Skill UI

This guide provides comprehensive, code-based solutions for diagnosing and fixing common errors encountered in Databricks environments. It covers critical issues such as cluster state failures, Spark OutOfMemory errors, Delta Lake concurrency conflicts, permission problems, and schema mismatches, offering tailored fixes in Python, SQL, and Bash.

This comprehensive guide details strategies for migrating enterprise data platforms to Databricks. It provides PySpark examples for discovery, schema mapping (converting types from Hadoop, Snowflake, etc.), and executing robust data transfer methods like SYNC, DEEP CLONE, and CTAS. Use this when moving data from legacy sources like Hadoop, Snowflake, or Redshift to a modern Delta Lake environment.

A comprehensive guide detailing optimization techniques for Databricks environments. It covers optimal cluster sizing for various workloads (ETL, ML, Streaming), advanced Spark configurations (AQE, memory management), and deep data optimization strategies for Delta Lake, including Z-Ordering and Liquid Clustering. It also provides methods for query plan analysis and join optimization.

Automated assistance for pyspark transformer work within data pipelines, activating when you request transformer patterns or best practices and covering ETL, data transformation, orchestration, and streaming scenarios.

Automates spark job creation within data pipelines by guiding etl, transformation, orchestration, and streaming workflows; includes industry best practices, code templates, and validation to deliver production-ready jobs.

This guide provides advanced patterns and best practices for optimizing Apache Spark jobs. It covers critical areas like effective partitioning, memory tuning, minimizing data skew, and optimizing different types of joins (broadcast, bucket). Use this when troubleshooting slow, large-scale data processing workflows or needing to scale Spark pipelines efficiently and reliably.

Automates Spark SQL optimizer tasks in data pipelines by guiding best practices, producing production-ready code and configurations, and validating ETL, transformation, orchestration, and streaming workflows.

Generates a single formula-driven Excel dashboard from CSV/tabular data, complete with KPI cards, charts, sparklines, and conditional formatting so analysts can instantly review metrics and share an executive-ready .xlsx output.

This guide provides comprehensive technical instructions for optimizing Palantir Foundry costs. Learn to use incremental transforms to drastically reduce compute rebuilds, right-size Spark profiles based on data volume, and migrate from expensive API polling to efficient, event-driven webhooks, ensuring both cost efficiency and system stability.

A structured guide for setting up a robust local development environment for Palantir Foundry data pipelines. This workflow enables users to run Python transforms and unit tests using PySpark against sample data, and mock the Foundry API client, ensuring fast, isolated iteration cycles before deploying code to the official Foundry environment. Ideal for data engineers and developers building complex ETL logic.

Execute arbitrary PySpark or pure Python code directly on Fabric Spark compute using the Livy API. This method ensures that no notebook artifacts are created or persisted, making it ideal for automated, ephemeral data transformations and ETL pipelines. It provides full read/write access to Delta lakehouse tables.