{"id":"arxiv/2605.09176","url":"https://arxiv.org/abs/2605.09176","tags":["cs.LG","cs.AI","math.OC"],"score":0,"title":"Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers","domain":"cs.LG","source":"arxiv","authors":["Aditya Ranganath"],"reviews":0,"subject":"cs.LG","summary":"Training large language models requires optimization algorithms that are not only statistically effective, but also computationally and memory efficient at extreme scale. Although Adam remains the dominant optimizer for large-scale language-model pretraining and fine-tuning, recent work has revisited nearly every component of the optimization stack: adaptive moment estimation, decoupled weight decay, memory footprint, curvature approximation, sign-based updates, large-batch stability, low-rank gradient structure, and matrix-wise orthogonalized updates. This survey reviews optimizer design…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-09","reactions":{}}