{"id":"arxiv/2603.04895","url":"https://arxiv.org/abs/2603.04895","tags":["stat.ML","cs.LG","math.OC"],"score":0,"title":"How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?","domain":"stat.ML","source":"arxiv","authors":["Kuo-Wei Lai","Guanghui Wang","Molei Tao","Vidya Muthukumar"],"reviews":0,"subject":"stat.ML","summary":"Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work (Vardi and Shamir, 2021) showed that the implicit bias does not exist in the worst-case, or corresponds exactly to the minimum-$\\ell_2$-norm interpolating…","category":"research","comments":0,"disputed":false,"division":"stat","mentions":0,"discussed":0,"published":"2026-03-05","reactions":{}}