Global Optimality Of Softmax Policy Gradient With Single Hidden Layer Neural Networks In The Mean-field Regime

Abstract

We study the problem of policy optimization for infinite-horizon discounted Markov Decision Processes with softmax policy and nonlinear function approximation trained with policy gradient algorithms. We concentrate on the training dynamics in the mean-field regime, modeling e.g., the behavior of wide single hidden layer neural networks, when exploration is encouraged through entropy regularization. The dynamics of these models is established as a Wasserstein gradient flow of distributions in parameter space. We further prove global optimality of the fixed points of this dynamics under mild conditions on their initialization.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyagazzi2020global

Related papers

• Convergence Of Policy Gradient For Entropy Regularized Mdps With Neural Network Approximation In The Mean-field Regime (2022)0.00
• On The Global Convergence Rates Of Decentralized Softmax Gradient Play In Markov Potential Games (2022)0.00
• On The Global Convergence Rates Of Softmax Policy Gradient Methods (2020)0.00
• Convergence And Price Of Anarchy Guarantees Of The Softmax Policy Gradient In Markov Potential Games (2022)0.00
• Fast Global Convergence Of Natural Policy Gradient Methods With Entropy Regularization (2020)0.00
• Softmax Policy Gradient Methods Can Take Exponential Time To Converge (2021)6.34
• Rethinking The Global Convergence Of Softmax Policy Gradient With Linear Function Approximation (2025)0.00
• Matryoshka Policy Gradient For Entropy-regularized RL: Convergence And Global Optimality (2023)0.00