{"id":"arxiv/2605.09213","url":"https://arxiv.org/abs/2605.09213","tags":["math.AP","cs.LG","math.PR"],"score":0,"title":"Kinetic theory for Transformers and the lost-in-the-middle phenomenon","domain":"math.AP","source":"arxiv","authors":["Mitia Duerinckx","Borjan Geshkovski","Stefano Rossi"],"reviews":0,"subject":"math.AP","summary":"We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-05-09","reactions":{}}