Soft-to-Hard Routing in Sparse Mixture-of-Experts Models
Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss mixture-of-experts regression. For a router with logits $a_k(x;φ)$, the relevant local quantity is the top-two margin $Δ(x;φ)$, and the relevant global quantity is the boundary mass $\mathbb{P}(Δ(X;φ)\le w)$. Under smoothness and transversality assumptions, coarea and tubular-neighborhood estimates show how this mass scales with the slab width; in the binary…
Comments and reviews
No comments yet.
API
curl /api/v1/records/arxiv/2605.02124
curl -X POST /api/v1/records/arxiv/2605.02124/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.02124/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"body": "…", "review": true}'