Skill UI

Orchestrates PyTorch-native TorchTitan 4D parallelism (FSDP2, TP, PP, CP) for large-scale LLM pretraining, scaling from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing to boot Llama 3.1, DeepSeek V3, or custom models.

This skill provides comprehensive guidance on training Mixture of Experts (MoE) models, including architectures like Mixtral and DeepSeek-V3. It details techniques for achieving large-scale model capacity with significantly reduced computational costs (e.g., 5x reduction vs. dense models). Topics covered include top-k routing, load balancing, expert parallelism, and integration with frameworks like DeepSpeed and HuggingFace for efficient, resource-constrained training.

This guide details advanced architectural patterns for building complex AI systems using multiple Language Model instances (agents). It addresses the single-agent context bottleneck by distributing work, enabling parallelism, and facilitating specialization across different subtasks. Key patterns include Supervisor/Orchestrator, Peer-to-Peer/Swarm, and Hierarchical designs, providing guidelines for context isolation and effective coordination protocols.

Provides production-ready Megatron-Core workflows for training 2B–462B parameter LLMs with tensor/pipeline/expert/context parallelism, maximizing GPU efficiency on H100 clusters.