The injectivity threshold of a Gaussian ReLU layer
We determine how wide a randomly initialized ReLU layer must be to distinguish every pair of inputs. For independent Gaussian weights and zero bias, as the dimensions grow at a fixed output-to-input ratio, the probability of global injectivity tends to zero below a sharp threshold and to one above it. We characterize this threshold exactly through a variational formula for the limiting minimum number of active neurons per input coordinate. The formula accounts for correlations among input directions at arbitrarily many levels. We prove this formula rigorously, establishing a prediction from…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2610.07492
curl -X POST /api/v1/records/arxiv/2610.07492/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2610.07492/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'