{"id":"arxiv/2602.01642","url":"https://arxiv.org/abs/2602.01642","tags":["cs.LG","cs.AI","math.OC","stat.CO","stat.ML"],"score":0,"title":"The Effect of Mini-Batch Noise on the Implicit Bias of Adam","domain":"cs.LG","source":"arxiv","authors":["Matias D. Cattaneo","Boris Shigida"],"reviews":0,"subject":"cs.LG","summary":"With limited high-quality data and growing compute, multi-epoch training is gaining back its importance across sub-areas of deep learning. Adam(W), versions of which are go-to optimizers for many tasks such as next token prediction, has two momentum hyperparameters $(β_1, β_2)$ controlling memory and one very important hyperparameter, batch size, controlling (in particular) the amount mini-batch noise. We introduce a theoretical framework to understand how mini-batch noise influences the implicit bias of memory in Adam (depending on $β_1$, $β_2$) towards sharper or flatter regions of the…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-02","reactions":{}}