On Adaptivity in Zeroth-Order Optimization

Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser, Christos Louizos

arXivarxiv/2605.03869May 5, 2026cs.LGscore 0

We investigate the effectiveness of adaptive zeroth-order (ZO) optimization for memory-constrained fine-tuning of large language models (LLMs). Contrary to prior claims, we show that adaptive ZO methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD, while incurring significant memory overhead. Our analysis reveals that in high dimensions, ZO gradients lack coordinate-wise heterogeneity, rendering adaptive mechanisms memory inefficient. Leveraging this insight, we propose MEAZO, a memory-efficient adaptive ZO optimizer that tracks only a single scalar for global step…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.03869
curl -X POST /api/v1/records/arxiv/2605.03869/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.03869/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'