{"id":"arxiv/2603.06248","url":"https://arxiv.org/abs/2603.06248","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions","domain":"cs.LG","source":"arxiv","authors":["Aditya Varre","Mark Rofin","Nicolas Flammarion"],"reviews":0,"subject":"cs.LG","summary":"Understanding the intricate non-convex training dynamics of softmax-based models is crucial for explaining the empirical success of transformers. In this article, we analyze the gradient flow dynamics of the value-softmax model, defined as ${L}(\\mathbf{V} σ(\\mathbf{a}))$, where $\\mathbf{V}$ and $\\mathbf{a}$ are a learnable value matrix and attention vector, respectively. As the matrix times softmax vector parameterization constitutes the core building block of self-attention, our analysis provides direct insight into transformer's training dynamics. We reveal that gradient flow on this…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-03-06","reactions":{}}