Entropy Regularization With Discounted Future State Distribution In Policy Gradient Methods

Abstract

The policy gradient theorem is defined based on an objective with respect to the initial distribution over states. In the discounted case, this results in policies that are optimal for one distribution over initial states, but may not be uniformly optimal for others, no matter where the agent starts from. Furthermore, to obtain unbiased gradient estimates, the starting point of the policy gradient estimator requires sampling states from a normalized discounted weighting of states. However, the difficulty of estimating the normalized discounted weighting of states, or the stationary state distribution, is quite well-known. Additionally, the large sample complexity of policy gradient methods is often attributed to insufficient exploration, and to remedy this, it is often assumed that the restart distribution provides sufficient exploration in these algorithms. In this work, we propose exploration in policy gradient methods based on maximizing entropy of the discounted future state distri

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyislam2019entropy

Related papers

• Marginalized State Distribution Entropy Regularization In Policy Optimization (2019)0.00
• Beyond Exact Gradients: Convergence Of Stochastic Soft-max Policy Gradient Methods With Entropy Regularization (2021)2.26
• A Temporal-difference Approach To Policy Gradient Estimation (2022)0.00
• Off-policy Policy Gradient With State Distribution Correction (2019)0.00
• Increasing Entropy To Boost Policy Gradient Performance On Personalization Tasks (2023)0.00
• Correcting Discount-factor Mismatch In On-policy Policy Gradient Methods (2023)0.00
• Analysis Of On-policy Policy Gradient Methods Under The Distribution Mismatch (2025)0.00
• Fast Global Convergence Of Natural Policy Gradient Methods With Entropy Regularization (2020)0.00