Optimal higher order convergence rates for the MUON optimizer
Source: arxiv.org/abs/2611.15825
Gradient descent optimization methods are the key tools to train deep neural networks in artificial intelligence systems. In practice, one typically employs suitable accelerated gradient descent optimization methods such as the momentum optimizer, the Adam optimizer, or the MUON optimizer proposed by Jordan et al. in 2024. In 1964 Polyak showed that, in a neighborhood of a local minimizer, the momentum optimizer converges with the rate $(\sqrtχ+1)^{-1}(\sqrtχ-1)$, which is strictly faster than the rate $(χ+1)^{-1}(χ-1)$ of the standard gradient descent optimizer, where $χ\in(1,\infty)$ is…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2611.15825
curl -X POST /api/v1/records/arxiv/2611.15825/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2611.15825/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'