Download

Skill UI

Browse and discover 5029+ curated skills

All Development Artificial Intelligence Design & Creative Product & Business Data Science Marketing Soft Skills Productivity Engineering Languages

Search Distributed Training , found 12 results

Default Newest Most Downloaded

DeepSpeed Training Assistant

Orchestra-Research/AI-Research-SKILLs

Provides expert guidance for DeepSpeed distributed training, covering ZeRO optimizations, pipeline parallelism, mixed precision, DeepNVMe I/O, and best practices for debugging or deploying large-scale models.

TorchTitan LLM Pretraining

distributed-llm-pretraining-torchtitan

Orchestra-Research/AI-Research-SKILLs

TorchTitan delivers PyTorch-native distributed LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), Float8, torch.compile, and checkpointing so you can scale Llama 3.1, DeepSeek V3, or custom models across 8 to 512+ GPUs.

Distributed Training Setup

distributed-training-setup

jeremylongshore/claude-code-plugins-plus-skills

Guides automated distributed training setups with step-by-step practices, production-ready configs, and validation checks for ML training workflows.

Accelerate Distributed Training

huggingface-accelerate

Orchestra-Research/AI-Research-SKILLs

Accelerate wraps PyTorch scripts with a four-line integration so you can launch DDP, DeepSpeed, FSDP, Megatron, or mixed-precision training on any hardware via a single command, handling device placement, sharding, and automatic config generation.

OpenRLHF High Performance Training

openrlhf-training

Orchestra-Research/AI-Research-SKILLs

OpenRLHF is a Ray-based RLHF framework that accelerates large-model PPO, GRPO, RLOO, and DPO training with vLLM inference and ZeRO-3 resource sharing, targeting distributed GPU clusters handling 7B-70B+ parameter models.

Lightning Training Suite

pytorch-lightning

Orchestra-Research/AI-Research-SKILLs

High-level PyTorch Lightning streamlines training loops with automatic distributed support (DDP/FSDP/DeepSpeed), callbacks, checkpointing, logging, and device management so you can scale from a laptop to multi-node clusters without boilerplate.

Ray Train Distributed Orchestration

Orchestra-Research/AI-Research-SKILLs

Ray Train orchestrates distributed PyTorch/TensorFlow/HuggingFace training from laptops to thousands of nodes, adding elastic scaling, hyperparameter search, and fault tolerance while keeping most code unchanged.

SkyPilot Cloud Orchestration

skypilot-multi-cloud-orchestration

Orchestra-Research/AI-Research-SKILLs

SkyPilot Multi-Cloud Orchestration codifies ML workload management by spinning up clusters across AWS/GCP/Azure, auto-optimizing costs, recovering spot instances, and unifying distributed GPU training and serving workflows.

Torchforge RL Training

torchforge-rl-training

Orchestra-Research/AI-Research-SKILLs

Torchforge RL Training documents Meta's PyTorch-native torchforge library, which separates algorithms from infrastructure so researchers can explore GRPO/SFT losses, manage Monarch actors, tap TorchTitan and vLLM for scalable distributed runs, and monitor KL/loss curves.

Megatron LLM Training

training-llms-megatron

Orchestra-Research/AI-Research-SKILLs

Megatron-Core orchestrates training of 2B–462B parameter LLMs with tensor/pipeline/context/expert parallelism, maximizing H100 MFU for production-grade distributed training pipelines.

Lightning Training Workflow

pytorch-lightning

K-Dense-AI/claude-scientific-skills

PyTorch Lightning organizes models into LightningModule, data modules, callbacks, logging, and Trainer orchestration for scalable multi-GPU/TPU training with distributed strategies and experiment tracking.

Hugging Face Transformers

K-Dense-AI/claude-scientific-skills

Load Hugging Face Transformers to run pipelines for text, vision, audio, and multimodal inference, customize model loading, tokenize inputs, and fine-tune with distributed training on domain-specific datasets.

1

Language