{"id":"arxiv/2605.06563","url":"https://arxiv.org/abs/2605.06563","tags":["cs.LG","hep-th"],"score":0,"title":"Criticality and Saturation in Orthogonal Neural Networks","domain":"cs.LG","source":"arxiv","authors":["Max Guillen","Jan E. Gerken"],"reviews":0,"subject":"cs.LG","summary":"It has been known for a long time that initializing weight matrices to be orthogonal instead of having i.i.d. Gaussian components can improve training performance. This phenomenon can be analyzed using finite-width corrections, where the infinite-width statistics are supplemented by a power series in $1/\\mathrm{width}$. In particular, recent empirical results by Day et al. show that the tensors appearing in this treatment stabilize for large depth, as opposed to the tensors of i.i.d.-initialized networks. In this article, we derive explicit layer-wise recursion relations for the tensors…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-05-07","reactions":{}}