Error whitening: Why Gauss-Newton outperforms Newton

Maricela Best McKay, Nathan P. Lawrence, Brian Wetton, R. Bhushan Gopaluni

arXivarxiv/2605.11316May 11, 2026cs.LGscore 0

The Gauss-Newton matrix is widely viewed as a positive semidefinite approximation of the Hessian, yet mounting empirical evidence shows that Gauss-Newton descent outperforms Newton's method. We adopt a function space perspective to analyze this phenomenon. We show that the generalized Gauss-Newton (GGN) matrix projects the Newton direction in function space onto the model's tangent space, while a Jacobian-only variant obtained by applying the least squares Gauss-Newton matrix to non-least squares losses projects the function space loss gradient onto this same tangent space. Both projections…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.11316
curl -X POST /api/v1/records/arxiv/2605.11316/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.11316/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'