Adaptive Trust Region Policy Optimization: Global Convergence And Faster Rates For Regularized Mdps

Abstract

Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspired by Conservative Policy Iteration (CPI). We show that the adaptive scaling mechanism used in TRPO is in fact the natural "RL version" of traditional trust-region methods from convex analysis. We first analyze TRPO in the planning setting, in which we have access to the model and the entire state space. Then, we consider sample-based TRPO and establish \(\tilde O(1/\sqrt\{N\})\) convergence rate to the global optimum. Importantly, the adaptive scaling mechanism allows us to analyze TRPO in regularized MDPs for which we prove fast rates of \(\tilde O(1/N)\), much like results in convex optimization. This is the first result in RL of better rates when regularizing the ins

Stats

• citations40
• S2 citations—
• github stars0
• HF likes0
• heat score12.10
• arxiv keyshani2019adaptive

Related papers

• Entrpo: Trust Region Policy Optimization Method With Entropy Regularization (2021)0.00
• Simple Policy Optimization (2024)0.00
• Embedding Safety Into RL: A New Take On Trust Region Methods (2024)0.00
• Neural Proximal/trust Region Policy Optimization Attains Globally Optimal Policy (2019)0.00
• Trust-pcl: An Off-policy Trust Region Method For Continuous Control (2017)0.00
• Multi-agent Trust Region Policy Optimization (2020)12.61
• Hindsight Trust Region Policy Optimization (2019)0.00
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization (2018)0.00