Why Policy Gradient Algorithms Work For Undiscounted Total-reward Mdps

Abstract

The classical policy gradient method is the theoretical and conceptual foundation of modern policy-based reinforcement learning (RL) algorithms. Most rigorous analyses of such methods, particularly those establishing convergence guarantees, assume a discount factor \(\gamma < 1\). In contrast, however, a recent line of work on policy-based RL for large language models uses the undiscounted total-reward setting with \(\gamma = 1\), rendering much of the existing theory inapplicable. In this paper, we provide analyses of the policy gradient method for undiscounted expected total-reward infinite-horizon MDPs based on two key insights: (i) the classification of the MDP states into recurrent and transient states is invariant over the set of policies that assign strictly positive probability to every action (as is typical in deep RL models employing a softmax output layer) and (ii) the classical state visitation measure (which may be ill-defined when \(\gamma = 1\)) can be replaced with a ne

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2025why

Related papers

• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00
• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Residual Policy Gradient: A Reward View Of Kl-regularized Objective (2025)0.00
• Revisiting Estimation Bias In Policy Gradients For Deep Reinforcement Learning (2023)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Variational Policy Gradient Method For Reinforcement Learning With General Utilities (2020)0.00
• Policy Gradient For Reinforcement Learning With General Utilities (2022)0.00