{"id":"arxiv/2609.36600","url":"https://arxiv.org/abs/2609.36600","tags":["math.OC","cs.AI","stat.ML"],"score":0,"title":"Second-Moment Stochastic Approximation Methods","domain":"math.OC","source":"arxiv","authors":["Tao Jiang","Lin Xiao"],"reviews":0,"subject":"math.OC","summary":"Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-09-29","reactions":{}}