One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Shows one-step gradient delay in asynchronous pipeline parallelism does not harm LLM pretraining convergence, enabling efficient GPU utilization at scale.