Rethinking The Global Convergence Of Softmax Policy Gradient With Linear Function Approximation

Abstract

Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning. In order to handle large state-action spaces, PG methods are typically used with function approximation. In this setting, the approximation error in modeling problem-dependent quantities is a key notion for characterizing the global convergence of PG methods. We focus on Softmax PG with linear function approximation (referred to as \(\texttt\{Lin-SPG\}\)) and demonstrate that the approximation error is irrelevant to the algorithm's global convergence even for the stochastic bandit setting. Consequently, we first identify the necessary and sufficient conditions on the feature representation that can guarantee the asymptotic global convergence of \(\texttt\{Lin-SPG\}\). Under these feature conditions, we prove that \(T\) iterations of \(\texttt\{Lin-SPG\}\) with a problem-specific learning rate result in an \(O(1/T)\) convergence to the optimal policy. Furthermore, we prove th

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylin2025rethinking

Related papers

• Softmax Policy Gradient Methods Can Take Exponential Time To Converge (2021)6.34
• Linear Convergence Of Entropy-regularized Natural Policy Gradient With Linear Function Approximation (2021)6.34
• On The Global Convergence Rates Of Softmax Policy Gradient Methods (2020)0.00
• An Alternate Policy Gradient Estimator For Softmax Policies (2021)0.00
• Symmetric (optimistic) Natural Policy Gradient For Multi-agent Learning With Parameter Convergence (2022)0.00
• On The Global Convergence Rates Of Decentralized Softmax Gradient Play In Markov Potential Games (2022)0.00
• Fast Global Convergence Of Natural Policy Gradient Methods With Entropy Regularization (2020)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00