Long-Time Behavior of Zero-Noise Mean-Field Adam
Adaptive methods such as Adam are the most commonly used methods for training neural networks, but their long-time behavior is not as well understood as that of non-adaptive methods. This is particularly true in the mean-field limit, where the dynamics of infinitely many neurons are described as a distribution over parameters which satisfies a self-consistency condition. In this paper, we study a deterministic continuous-time model of Adam in the mean-field setting. Under a condition relating the averaging rates of the momentum and second-moment variables, the size of the gradients, and a…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2611.01658
curl -X POST /api/v1/records/arxiv/2611.01658/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2611.01658/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'