{"id":"arxiv/2607.17620","url":"https://arxiv.org/abs/2607.17620","tags":["cs.LG","cs.CL","math.OC"],"score":0,"title":"PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer","domain":"cs.LG","source":"arxiv","authors":["Nikhil Ghosh","Tetiana Parshakova","Robert M. Gower"],"reviews":0,"subject":"cs.LG","summary":"Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-07-20","reactions":{}}