{"id":"arxiv/2602.03386","url":"https://arxiv.org/abs/2602.03386","tags":["cs.LG","cs.AI","math.OC"],"score":0,"title":"An Approximate Ascent Approach To Prove Convergence of PPO","domain":"cs.LG","source":"arxiv","authors":["Leif Doering","Daniel Schmidt","Moritz Melcher","Sebastian Kassing","Benedikt Wille","Tilman Aach","Simon Weissmann"],"reviews":0,"subject":"cs.LG","summary":"Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-02-03","reactions":{}}