Training Infinitely Deep and Wide Transformers

Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya, Gabriel Peyré

arXivarxiv/2605.17660May 17, 2026math.OCscore 0

Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited. This paper develops a rigorous mathematical framework for analyzing gradient-based training of transformers in the mean-field regime, where both the depth (number of layers) and width (number of attention heads) tend to infinity. While ResNet training can be understood as controlling a neural ODE, transformer training corresponds to controlling a neural PDE, due to the coupling of multiple token distributions through the attention…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.17660
curl -X POST /api/v1/records/arxiv/2605.17660/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.17660/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'