Linearized Attention Cannot Enter the Kernel Regime at Any Practical Width

Jose Marie Antonio Miñoza, Paulo Mario P. Medina, Sebastian C. Ibañez

arXivarxiv/2603.13085Mar 13, 2026cs.LGscore 0

Understanding whether attention mechanisms converge to the kernel regime is foundational to the validity of influence functions for transformer accountability. Exact NTK characterization of softmax attention is precluded by its exponential nonlinearity; linearized attention is the canonical tractable proxy and the object of study here. This paper establishes that even this proxy does not converge to its NTK limit at any practical width, revealing a fundamental trade-off in the learning dynamics of attention. An exact correspondence is established between parameter-free linearized attention…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2603.13085
curl -X POST /api/v1/records/arxiv/2603.13085/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2603.13085/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'