{"id":"arxiv/2604.09940","url":"https://arxiv.org/abs/2604.09940","tags":["cs.AI","cs.LG","math.OC"],"score":0,"title":"New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework","domain":"cs.AI","source":"arxiv","authors":["Shaocong Ma","Peiran Yu","Heng Huang"],"reviews":0,"subject":"cs.AI","summary":"Fine-tuning Large Language Models (LLMs) typically involves either full fine-tuning, which updates all model parameters, or Parameter-Efficient Fine-Tuning (PEFT), which adjusts a small subset of parameters. However, both approaches have inherent limitations: full fine-tuning is computationally expensive, while PEFT often struggles to learn new knowledge and exhibits suboptimal performance. To overcome these issues, we propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth-order and first-order optimization methods. To…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-04-10","reactions":{}}