{"id":"arxiv/2603.16535","url":"https://arxiv.org/abs/2603.16535","tags":["cs.LG","math.OC","stat.ML"],"score":0,"title":"SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds","domain":"cs.LG","source":"arxiv","authors":["Viktor Stein","Wuchen Li","Gabriele Steidl"],"reviews":0,"subject":"cs.LG","summary":"Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein-$2$-type metrics. We extend this viewpoint by introducing accelerated attention blocks derived from inertial Nesterov-type dynamics on density spaces. In our proposed architecture, tokens carry both spatial (feature) and velocity variables. The time discretization…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-03-17","reactions":{}}