{"id":"arxiv/2602.13659","url":"https://arxiv.org/abs/2602.13659","tags":["cs.LG","math.OC"],"score":0,"title":"Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling","domain":"cs.LG","source":"arxiv","authors":["Valery Parfenov","Grigoriy Evseev","Andrey Veprikov","Nikolay Bushkov","Stanislav Moiseev","Aleksandr Beznosikov"],"reviews":0,"subject":"cs.LG","summary":"Fine-tuning large pretrained language models (LLMs) is a cornerstone of modern NLP, yet its growing memory demands (driven by backpropagation and large optimizer States) limit deployment in resource-constrained settings. Zero-order (ZO) methods bypass backpropagation by estimating directional derivatives from forward evaluations, offering substantial memory savings. However, classical ZO estimators suffer from high variance and an adverse dependence on the parameter dimensionality $d$, which has constrained their use to low-dimensional problems. In this work, we propose a policy-driven ZO…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-14","reactions":{}}