{"id":"arxiv/2601.22257","url":"https://arxiv.org/abs/2601.22257","tags":["cs.LG","hep-th"],"score":0,"title":"Symmetry Breaking in Transformers for Efficient and Interpretable Training","domain":"cs.LG","source":"arxiv","authors":["Eva Silverstein","Daniel Kunin","Vasudev Shyam"],"reviews":0,"subject":"cs.LG","summary":"The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol that inserts a preferred direction into this rotational space through batchwise-sampled, unlearned query and value biases. This modification has two theoretically motivated and empirically validated consequences. First, it can substantially improve the performance of simple, memory-efficient optimizers, narrowing -- and in some cases closing -- the gap to…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-01-29","reactions":{}}