Analysis Of On-policy Policy Gradient Methods Under The Distribution Mismatch

Abstract

Policy gradient methods are one of the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical successes, many state-of-the-art policy gradient algorithms for discounted problems deviate from the theoretical policy gradient theorem due to the existence of a distribution mismatch. In this work, we analyze the impact of this mismatch on policy gradient methods. Specifically, we first show that in the case of tabular parameterizations, the biased gradient induced by the mismatch still yields a valid first-order characterization of global optimality. Then, we extend this analysis to more general parameterizations by deriving explicit bounds on both the state distribution mismatch and the resulting gradient mismatch in episodic and continuing MDPs, which are shown to vanish at least linearly as the discount factor approaches one. Building on these bounds, we further establish guarantees for the biased policy gradient iterates, showing that

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2025analysis

Related papers

• Off-policy Policy Gradient With State Distribution Correction (2019)0.00
• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• Correcting Discount-factor Mismatch In On-policy Policy Gradient Methods (2023)0.00
• Revisiting Estimation Bias In Policy Gradients For Deep Reinforcement Learning (2023)0.00
• Policy Gradient Method For Robust Reinforcement Learning (2022)0.00
• On The Convergence Of Discounted Policy Gradient Methods (2022)0.00
• Why Policy Gradient Algorithms Work For Undiscounted Total-reward Mdps (2025)0.00
• A Temporal-difference Approach To Policy Gradient Estimation (2022)0.00