{"id":"arxiv/2605.07959","url":"https://arxiv.org/abs/2605.07959","tags":["cs.LG","math.FA","math.PR"],"score":0,"title":"Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRA","domain":"cs.LG","source":"arxiv","authors":["Zhengkai Sun","Dibyakanti Kumar","Alejandro F Frangi","Anirbit Mukherjee","Mingfei Sun"],"reviews":0,"subject":"cs.LG","summary":"Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications. Further, for large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size trade-off. In this work, via a unified framework we rigorously establish trainability of such models under stochastic methods. We prove that for a class of mild regularizations, the empirical regression loss on a attention layer and LoRA on a shallow…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-08","reactions":{}}