[Must Read: Beginners] What I did read for learning the basics of Distributed Training for LLMs From Scratch: Building a distributed training framework from first principles by Umar Jamil on YT Horovod: Fast and Easy Distributed Deep Learning in TensorFlow PyTorch Distributed: Experiences on Accelerating Data Parallel Training Large Scale Distributed Deep Networks ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel Link to the above resources here!   submitted by   /u/East-Muffin-6472 [link]   [comments]