Stochastic Gradient Methods with Online Scaling

Wanyu Zhang, Wenzhi Gao, Yinyu Ye, Madeleine Udell

arXivarxiv/2609.11751Sep 10, 2026math.OCscore 0

This paper introduces Stochastic Online Scaled Gradient Methods (SOSGM), a generalization of the recently developed adaptive preconditioning framework in arXiv:2505.23081 and arXiv:2509.11007 to stochastic optimization. Under standard assumptions, we establish convergence guarantees for SOSGM using large batchsize or variance reduction. SOSGM is compatible with popular diagonal and/or low-rank preconditioners as well as heavy-ball momentum, while maintaining memory and computation cost comparable to Adam. Extensive numerical experiments demonstrate the strong empirical performance of SOSGM.…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2609.11751
curl -X POST /api/v1/records/arxiv/2609.11751/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2609.11751/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'