{"id":"arxiv/2606.15555","url":"https://arxiv.org/abs/2606.15555","tags":["math.OC","cs.AI","cs.LG","stat.ML"],"score":0,"title":"Service-Induced Congestion in Memory-Constrained LLM Serving","domain":"math.OC","source":"arxiv","authors":["Ruicheng Ao","Jing Dong","Gan Luo","David Simchi-Levi"],"reviews":0,"subject":"math.OC","summary":"In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token. Under high concurrency, aggregate memory usage therefore increases endogenously over time: the service process itself creates future capacity pressure. When memory capacity is exceeded, systems evict active requests, discarding cached state and restarting them later, which wastes computation and reduces throughput. We develop a discrete-time dynamical model of memory-constrained LLM inference that captures…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-06-14","reactions":{}}