Policy Gradient In Partially Observable Environments: Approximation And Convergence

Abstract

Policy gradient is a generic and flexible reinforcement learning approach that generally enjoys simplicity in analysis, implementation, and deployment. In the last few decades, this approach has been extensively advanced for fully observable environments. In this paper, we generalize a variety of these advances to partially observable settings, and similar to the fully observable case, we keep our focus on the class of Markovian policies. We propose a series of technical tools, including a novel notion of advantage function, to develop policy gradient algorithms and study their convergence properties in such environments. Deploying these tools, we generalize a variety of existing theoretical guarantees, such as policy gradient and convergence theorems, to partially observable domains, those which also could be carried to more settings of interest. This study also sheds light on the understanding of policy gradient approaches in real-world applications which tend to be partially observa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyazizzadenesheli2018policy

Related papers

• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• A Study Of Policy Gradient On A Class Of Exactly Solvable Models (2020)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• Convergence And Optimality Of Policy Gradient Methods In Weakly Smooth Settings (2021)3.58
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• On The Linear Convergence Of Natural Policy Gradient Algorithm (2021)0.00
• Policy Gradient Algorithms Implicitly Optimize By Continuation (2023)0.00
• Smoothing Policies And Safe Policy Gradients (2019)7.50