Setup Parallelism

• [Megatron-LM][Model Parallelism][Global Distributed Group] Questions
• [Megatron-LM][Tensor Parallelism] Questions
• [Megatron-LM][Pipeline Parallelism] Questions
• [Megatron-LM][Data Parallelism] Questions
• [OSLO][Setup Parallelism][1] Questions
• [OSLO][Setup Parallelism][2] Questions

Tensor Parallelism

• [Megatron-LM][Distributed Communication in Tensor Parallelism] Questions
• [Megatron-LM][Model Parallelism][Parallel Linear] Questions
• [Megatron-LM][Model Parallelism][Parallel Self-Attention] Questions
• [Megatron-LM][Model Parallelism][Parallel Embedding] Questions
• [Megatron-LM][Model Parallelism][Parallel Embedding][2] Questions
• [Megatron-LM][Model Parallelism][Parallel Cross Entropy][1] Questions
• [OSLO][Tensor parallelism][Turn any 🤗 transformers model into a tensor parallel model][1] Questions
• [OSLO][Tensor parallelism][Turn any 🤗 transformers model into a tensor parallel model][2] Questions