{"id":"arxiv/2605.17778","url":"https://arxiv.org/abs/2605.17778","tags":["math.ST","cs.LG","stat.ME","stat.ML"],"score":0,"title":"Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models","domain":"math.ST","source":"arxiv","authors":["Radu Lecoiu","Debarghya Mukherjee","Pragya Sur"],"reviews":0,"subject":"math.ST","summary":"Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with $s$ spikes, $s$-step self-distillation achieves optimal performance among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that $s$ steps are necessary for…","category":"research","comments":0,"disputed":false,"division":"math","mentions":0,"discussed":0,"published":"2026-05-18","reactions":{}}