{"id":"arxiv/2602.22271","url":"https://arxiv.org/abs/2602.22271","tags":["cs.LG","math.PR","math.ST"],"score":0,"title":"Support Tokens, Stability Margins, and a New Foundation for Robust LLMs","domain":"cs.LG","source":"arxiv","authors":["Deepak Agarwal","Dhyey Dharmendrakumar Mavani","Suyash Gupta","Karthik Sethuraman","Tejas Dharamsi"],"reviews":0,"subject":"cs.LG","summary":"Self-attention is usually described as a flexible, content-adaptive way to mix a token with information from its past. We reinterpret causal self-attention transformers, the backbone of modern foundation models, within a probabilistic framework, much as classical PCA is extended to probabilistic PCA. This reformulation reveals a key structural consequence of the underlying change of variables: a barrier constraint emerges on the parameters of self-attention. The resulting geometry exposes a degeneracy boundary where the attention-induced mapping becomes locally ill-conditioned, yielding a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-25","reactions":{}}