{"id":"arxiv/2604.01978","url":"https://arxiv.org/abs/2604.01978","tags":["math.PR","cs.LG","stat.ML"],"score":0,"title":"Homogenized Transformers","domain":"math.PR","source":"arxiv","authors":["Hugo Koubbi","Borjan Geshkovski","Philippe Rigollet"],"reviews":0,"subject":"math.PR","summary":"We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth as a time variable, the residual stream defines a discrete-time interacting particle system on the unit sphere. We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivial homogenized limit. Depending on the scaling, the limit is either deterministic or stochastic with common noise; in the mean-field regime, the latter leads to a…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-04-02","reactions":{}}