Spectral Selection in Symmetric Self-Attention Dynamics
We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a gradient-flow structure and an exact reformulation in the eigenbasis of $V$, revealing a spectral mode-selection mechanism. We show that the dynamics exhibits two distinct asymptotic scenarios: homogeneous alignment toward the dominant eigendirection when one positive eigenvalue strictly dominates all others in modulus, and sign-split polarization toward the most…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2604.26085
curl -X POST /api/v1/records/arxiv/2604.26085/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2604.26085/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'