Convergence Proof For Actor-critic Methods Applied To PPO And RUDDER

Abstract

We prove under commonly used assumptions the convergence of actor-critic reinforcement learning algorithms, which simultaneously learn a policy function, the actor, and a value function, the critic. Both functions can be deep neural networks of arbitrary complexity. Our framework allows showing convergence of the well known Proximal Policy Optimization (PPO) and of the recently introduced RUDDER. For the convergence proof we employ recently introduced techniques from the two time-scale stochastic approximation theory. Our results are valid for actor-critic methods that use episodic samples and that have a policy that becomes more greedy during learning. Previous convergence proofs assume linear function approximation, cannot treat episodic examples, or do not consider that policies become greedy. The latter is relevant since optimal policies are typically deterministic.

Stats

• citations35
• S2 citations—
• github stars0
• HF likes0
• heat score11.67
• arxiv keyholzleitner2020convergence

Related papers

• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Finite-time Analysis Of Natural Actor-critic For Pomdps (2022)0.00
• An Approximate Policy Iteration Viewpoint Of Actor-critic Algorithms (2022)2.26
• Global Convergence Of The ODE Limit For Online Actor-critic Algorithms In Reinforcement Learning (2021)0.00
• On The Sample Complexity Of Actor-critic Method For Reinforcement Learning With Function Approximation (2019)11.49
• Modified Actor-critics (2019)0.00
• Provable Benefits Of Actor-critic Methods For Offline Reinforcement Learning (2021)0.00
• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00