{"id":"arxiv/2609.28029","url":"https://arxiv.org/abs/2609.28029","tags":["math.NA","cs.CL","cs.LG"],"score":0,"title":"Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison","domain":"math.NA","source":"arxiv","authors":["Rahul Krishnan","Volker Schulz"],"reviews":0,"subject":"math.NA","summary":"The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level.…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-09-23","reactions":{}}