On The Global Optimality Of Policy Gradient Methods In General Utility Reinforcement Learning

Abstract

Reinforcement learning with general utilities (RLGU) offers a unifying framework to capture several problems beyond standard expected returns, including imitation learning, pure exploration, and safe RL. Despite recent fundamental advances in the theoretical analysis of policy gradient (PG) methods for standard RL and recent efforts in RLGU, the understanding of these PG algorithms and their scope of application in RLGU still remain limited. In this work, we establish global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure. In the tabular setting, we provide global optimality results using a new proof technique building on recent theoretical developments on the convergence of PG methods for standard RL using gradient domination. Our proof technique opens avenues for analyzing policy parameterizations beyond the direct policy parameterization for RLGU. In addition, we provide global optimality

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybarakat2024on

Related papers

• Policy Gradient For Reinforcement Learning With General Utilities (2022)0.00
• Variational Policy Gradient Method For Reinforcement Learning With General Utilities (2020)0.00
• Smoothing Policies And Safe Policy Gradients (2019)7.50
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• Last-iterate Global Convergence Of Policy Gradients For Constrained Reinforcement Learning (2024)0.00
• Residual Policy Gradient: A Reward View Of Kl-regularized Objective (2025)0.00