Skill UI

Orchestrates PyTorch-native TorchTitan 4D parallelism (FSDP2, TP, PP, CP) for large-scale LLM pretraining, scaling from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing to boot Llama 3.1, DeepSeek V3, or custom models.

HuggingFace Accelerate adds distributed support to any PyTorch script with just four lines, providing a unified API for DDP, DeepSpeed, FSDP, and Megatron, plus automatic device placement, mixed precision, and simple launch/config commands for fast prototyping.

Guides adding PyTorch FSDP2’s fully_shard integration to training loops, covering correct distributed init, bottom-up sharding, mixed precision/offload policies, DTensor-aware optimizers, and distributed checkpointing so oversized models or DeviceMesh sharding goals work reliably.

PyTorch Lightning is a high-level framework built on PyTorch that significantly reduces boilerplate code in deep learning training loops. It automatically handles device management, distributed training (DDP, FSDP, DeepSpeed), mixed precision, and advanced callbacks. It allows researchers to scale models from local machines to supercomputers with minimal code changes, enabling clean, scalable, and best-practice deep learning implementations.

PyTorch Lightning is a deep learning framework that organizes PyTorch code to eliminate boilerplate while maintaining flexibility. It automates training workflows, supports multi-GPU/TPU orchestration, and implements best practices for neural network training. Features include LightningModules, Trainers, data pipelines, callbacks, logging (W&B, TensorBoard), and distributed training strategies like DDP and FSDP for scalable training.

This skill covers setting up infrastructure for fine-tuning Large Language Models (LLMs) using techniques like QLoRA, LoRA, and full fine-tuning. It includes configuring distributed training with DeepSpeed or FSDP, utilizing frameworks such as Hugging Face TRL and Axolotl, and implementing alignment methods like RLHF and DPO. Suitable for domain-specific model adaptation and production deployment.

Add distributed training to any PyTorch script in just 4 lines with HuggingFace Accelerate. Unified API for DDP, DeepSpeed, FSDP and Megatron, with automatic device placement and FP16/BF16/FP8 mixed precision.

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP, covering parameter sharding, mixed precision, CPU offloading, FSDP2, and large-scale distributed training.

High-level PyTorch framework that simplifies training loops, automates distributed training with DDP/FSDP/DeepSpeed, and adds callbacks, logging, and checkpointing with minimal boilerplate.