Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics

Raphael Bernas, Fanny Jourdan, Antonin Poché, Céline Hudelot

arXivarxiv/2604.08764Apr 9, 2026cs.CLscore 0

Since their introduction, Transformer architectures have dominated Natural Language Processing (NLP). However, recent research has highlighted an inherent anisotropy phenomenon in these models, presenting a significant challenge to their geometric interpretation. Previous theoretical studies on this phenomenon are rarely grounded in the underlying representation geometry. In this paper, we extend them by deriving geometric arguments for how frequency-biased sampling attenuates curvature visibility and why training preferentially amplify tangent directions. Empirically, we then use…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2604.08764
curl -X POST /api/v1/records/arxiv/2604.08764/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2604.08764/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'