Fault tolerant distributed training on Amazon EKS using NVRx
Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.
Contenu original affiche; la traduction localisee n'est pas encore disponible.
Ce qui s'est passé
Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.
Pourquoi c'est important
The proceeding may create legal precedent, financial exposure or operating constraints for NVIDIA, Amazon.
Entités concernées
Voir les preuves
1 articles · 1 publication d'origine · 1 independantes
- AWS Machine Learning BlogSource primaire · Confirme · EN · 100%Fault tolerant distributed training on Amazon EKS using NVRx ↗
Affirmations
- Fault tolerant distributed training on Amazon EKS using NVRx Observé
Divergences
Aucune divergence importante détectée dans les preuves disponibles.
Chronologie
- Premier signalement
Mouvement de marché suivant l'événement
La réaction du marché n'est pas encore disponible pour cet actif et cette fenêtre.