{"id":"arxiv/2604.13656","url":"https://arxiv.org/abs/2604.13656","tags":["cs.LG","cs.AI","math.ST","stat.ML"],"score":0,"title":"Ordinary Least Squares is a Special Case of Transformer","domain":"cs.LG","source":"arxiv","authors":["Xiaojun Tan","Yuchen Zhao"],"reviews":0,"subject":"cs.LG","summary":"The statistical essence of the Transformer architecture has long remained elusive: Is it a universal approximator, or a neural network version of known computational algorithms? Through rigorous algebraic proof, we show that the latter better describes Transformer's basic nature: Ordinary Least Squares (OLS) is a special case of the single-layer Linear Transformer. Using the spectral decomposition of the empirical covariance matrix, we construct a specific parameter setting where the attention mechanism's forward pass becomes mathematically equivalent to the OLS closed-form projection. This…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-04-15","reactions":{}}