Gradient descent with exponentially increasing stepsizes and restarts
Let $f:\mathbb{R}^d \rightarrow \mathbb{R}$. We consider gradient descent $x_{n+1} = x_n - τ_n \nabla f(x_n)$, where the stepsize $τ_n = τ\cdot e^{rn}$ is exponentially growing (with $τ> 0$ and $0 < r \ll 1$). This diverges for almost all initial values. We show that restarting the algorithm whenever $\|x_{n+1} - x_n\| \geq e^r\|x_n - x_{n-1}\|$ has good properties: it works very well in practice; we determine the limiting convergence rate in the case of convergence to a non-degenerate local minimum: it improves on classic gradient descent even though computational cost is comparable. The…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2607.06314
curl -X POST /api/v1/records/arxiv/2607.06314/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2607.06314/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'