Hugging Face Accelerate
Add distributed training to any PyTorch script in a few lines with Hugging Face Accelerate. Unified API over DeepSpeed, FSDP, and DDP with automatic device placement and mixed precision (FP16/BF16/FP8).
This skill scales PyTorch training with Hugging Face Accelerate. It wraps an existing training loop to run on multi-GPU and multi-node setups, switches between DeepSpeed, FSDP, DDP, and Megatron backends through one API, enables mixed precision and automatic device placement, and generates the interactive config plus a single launch command.
huggingface accelerate distributed-training pytorch deepspeed
When to use
Use when you need to scale PyTorch training across GPUs or nodes, enable mixed precision, or add DeepSpeed/FSDP without rewriting your training loop.
Examples
Make training multi-GPU
Wrap a PyTorch loop
Add Hugging Face Accelerate to my PyTorch training script so it runs across 4 GPUs with BF16 mixed precision
Configure DeepSpeed
Fit a large model in memory
Set up Accelerate with DeepSpeed ZeRO-3 so I can fine-tune a 13B model on limited GPU memory