{"id":"arxiv/2601.10274","url":"https://arxiv.org/abs/2601.10274","tags":["cs.LG","cs.AI","cs.IT","cs.NI","math.OC"],"score":0,"title":"Queueing-Aware Optimization of Reasoning Tokens for Accuracy-Latency Trade-offs in LLM Servers","domain":"cs.LG","source":"arxiv","authors":["Emre Ozbas","Melih Bastopcu"],"reviews":0,"subject":"cs.LG","summary":"We consider a single large language model (LLM) server that serves a heterogeneous stream of queries belonging to $N$ distinct task types. Queries arrive according to a Poisson process, and each type occurs with a known prior probability. For each task type, the server allocates a fixed number of internal thinking tokens, which determines the computational effort devoted to that query. The token allocation induces an accuracy-latency trade-off: the service time follows an approximately affine function of the allocated tokens, while the probability of a correct response exhibits diminishing…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-01-15","reactions":{}}