{"id":"arxiv/2602.11246","url":"https://arxiv.org/abs/2602.11246","tags":["cs.LG","cs.AI","cs.CL","cs.IT","math.CO"],"score":0,"title":"How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?","domain":"cs.LG","source":"arxiv","authors":["Nikhil Garg","Jon Kleinberg","Kenny Peng"],"reviews":0,"subject":"cs.LG","summary":"We introduce a mathematical framework for the linear representation hypothesis (LRH), which asserts that intermediate layers of language models store features linearly. We separate the hypothesis into two claims: linear representation (features are linearly embedded in neuron activations) and linear accessibility (features can be linearly decoded). We then ask: How many neurons $d$ suffice to both linearly represent and linearly access $m$ features? Classical results in compressed sensing imply that for $k$-sparse inputs, $d = O(k\\log (m/k))$ suffices if we allow non-linear decoding…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-11","reactions":{}}