{"id":"arxiv/2605.07772","url":"https://arxiv.org/abs/2605.07772","tags":["cs.LG","math.AP","math.DS","math.OC"],"score":0,"title":"Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers","domain":"cs.LG","source":"arxiv","authors":["Noboru Isobe","Daisuke Inoue","Masaaki Imaizumi"],"reviews":0,"subject":"cs.LG","summary":"Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing analyses treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under $L^2$ regularization. We identify a training-induced phase of the inference…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-08","reactions":{}}