{"id":"arxiv/2602.03702","url":"https://arxiv.org/abs/2602.03702","tags":["cs.LG","cs.AI","math.OC","stat.ML"],"score":0,"title":"Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging","domain":"cs.LG","source":"arxiv","authors":["Alexandru Meterez","Pranav Ajit Nair","Depen Morwani","Cengiz Pehlevan","Sham Kakade"],"reviews":0,"subject":"cs.LG","summary":"Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretraining recipes are not anytime: they rely on horizon-dependent learning rate schedules and extensive tuning under a fixed compute budget. In this work, we provide a theoretical analysis demonstrating the existence of anytime learning schedules for overparameterized linear regression, and we highlight the central role of weight averaging - also known as model merging - in achieving the minimax convergence rates of…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-03","reactions":{}}