Does Weight Decay Enhance Training Stability?

Marius Saether, Amir Kolic, Tomaso Poggio, Pierfrancesco Beneventano

arXivarxiv/2605.16622May 15, 2026cs.LGscore 0

In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the \emph{Edge of Stability} (EoS). We show that weight decay robustly slows *progressive…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.16622
curl -X POST /api/v1/records/arxiv/2605.16622/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.16622/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'