Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression

Mingsong Yan, Dongyang Li, Charles Kulick, Sui Tang

arXivarxiv/2605.08475May 8, 2026cs.LGscore 0

In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass. Under bounded-data assumptions, we construct a single-head transformer whose forward pass approximately implements \textit{preconditioned Richardson iteration} on the associated kernel system. The construction uses $O(\log(1/ε))$ blocks and MLP width $O(\sqrt{N/ε})$ to achieve $ε$-accurate prediction for prompts of length $N$. Our construction reveals a…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.08475
curl -X POST /api/v1/records/arxiv/2605.08475/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.08475/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'