{"id":"arxiv/2606.10469","url":"https://arxiv.org/abs/2606.10469","tags":["math.OC","stat.ML"],"score":0,"title":"A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training","domain":"math.OC","source":"arxiv","authors":["Cheng Huan","Hongfwei Yuan"],"reviews":0,"subject":"math.OC","summary":"This paper develops a mean-field theory for a simplified single-layer causal multi-head self-attention model trained by cross-entropy minimization. Each attention head is treated as a particle in parameter space, and the empirical law of the heads is used as the large-head state variable. In the infinite-head limit, the averaged attention logits define a risk functional on probability measures, whose first variation generates a nonlinear Wasserstein gradient-flow equation. Unlike classical mean-field analyses of shallow networks that often focus on square-loss regression, the present model…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-06-09","reactions":{}}