{"id":"arxiv/2605.08475","url":"https://arxiv.org/abs/2605.08475","tags":["cs.LG","cs.AI","math.NA","math.OC"],"score":0,"title":"Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression","domain":"cs.LG","source":"arxiv","authors":["Mingsong Yan","Dongyang Li","Charles Kulick","Sui Tang"],"reviews":0,"subject":"cs.LG","summary":"In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass. Under bounded-data assumptions, we construct a single-head transformer whose forward pass approximately implements \\textit{preconditioned Richardson iteration} on the associated kernel system. The construction uses $O(\\log(1/ε))$ blocks and MLP width $O(\\sqrt{N/ε})$ to achieve $ε$-accurate prediction for prompts of length $N$. Our construction reveals a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-08","reactions":{}}