Skill UI

Orchestrates PyTorch-native TorchTitan 4D parallelism (FSDP2, TP, PP, CP) for large-scale LLM pretraining, scaling from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing to boot Llama 3.1, DeepSeek V3, or custom models.

Execute complex and compute-intensive workloads directly on fully managed Hugging Face cloud infrastructure. This skill eliminates the need for local setup, allowing users to run jobs using cloud CPUs, GPUs, or TPUs. Ideal for large-scale data processing, batch inference, reproducible model training, and scheduled automation tasks, while persisting results to the Hugging Face Hub.

This skill provides comprehensive guidance on training Mixture of Experts (MoE) models, including architectures like Mixtral and DeepSeek-V3. It details techniques for achieving large-scale model capacity with significantly reduced computational costs (e.g., 5x reduction vs. dense models). Topics covered include top-k routing, load balancing, expert parallelism, and integration with frameworks like DeepSpeed and HuggingFace for efficient, resource-constrained training.

torchforge is Meta's advanced PyTorch-native library designed for agentic Reinforcement Learning (RL). It fundamentally separates core RL algorithms from complex distributed infrastructure concerns. This allows researchers to rapidly prototype and experiment with novel RL methods (like GRPO) using clean, PyTorch-native abstractions, while automatically handling large-scale tasks like distributed training across thousands of GPUs using Monarch and TorchTitan.

This skill guides the secure handling and lifecycle management of large-scale data artifacts, including training datasets, model weights, and checkpoint snapshots, within GPU cloud environments. It provides best practices for data pipelines, covering secure import via Kubernetes PVCs, compliant export, and stringent data validation, ensuring adherence to encryption standards (AES-256) and regulatory requirements (PII masking, RBAC).

Expert guidance for implementing distributed training with DeepSpeed, covering ZeRO optimization stages, pipeline parallelism, mixed precision (FP16/BF16/FP8), and I/O optimizations like DeepNVMe. Helps with debugging, API usage, and best practices for large-scale model training on NVIDIA GPUs.

Megatron-Core is a production-ready framework for training large language models ranging from 2B to 462B parameters. It utilizes advanced parallelism strategies like tensor, pipeline, and expert parallelism to maximize GPU efficiency on NVIDIA H100 hardware. Ideal for scaling models beyond 1B parameters with high MFU performance.

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP, covering parameter sharding, mixed precision, CPU offloading, FSDP2, and large-scale distributed training.

Learn to train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. This skill covers sparse architectures like Mixtral 8x7B, reducing compute costs by 5x compared to dense models. It includes routing mechanisms, load balancing, expert parallelism, and inference optimization for scaling large-scale AI models efficiently.