Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners

Wait 5 sec.

[Must Read: Beginners] What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch: Building a distributed training framework from first principles by Umar Jamil on YT Horovod: Fast and Easy Distributed Deep Learning in TensorFlow PyTorch Distributed: Experiences on Accelerating Data Parallel Training Large Scale Distributed Deep Networks ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel Link to the above resources here Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)   submitted by   /u/East-Muffin-6472 [link]   [comments]