Marginalized State Distribution Entropy Regularization In Policy Optimization

Abstract

Entropy regularization is used to get improved optimization performance in reinforcement learning tasks. A common form of regularization is to maximize policy entropy to avoid premature convergence and lead to more stochastic policies for exploration through action space. However, this does not ensure exploration in the state space. In this work, we instead consider the distribution of discounted weighting of states, and propose to maximize the entropy of a lower bound approximation to the weighting of a state, based on latent space state representation. We propose entropy regularization based on the marginal state distribution, to encourage the policy to have a more uniform distribution over the state space for exploration. Our approach based on marginal state distribution achieves superior state space coverage on complex gridworld domains, that translate into empirical gains in sparse reward 3D maze navigation and continuous control domains compared to entropy regularization with sto

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyislam2019marginalized

Related papers

• Understanding The Impact Of Entropy On Policy Optimization (2018)0.00
• Policy Optimization Reinforcement Learning With Entropy Regularization (2019)0.00
• Entropy Regularization With Discounted Future State Distribution In Policy Gradient Methods (2019)0.00
• Entropy Regularization For Mean Field Games With Learning (2020)0.00
• Optimal Scheduling Of Entropy Regulariser For Continuous-time Linear-quadratic Reinforcement Learning (2022)4.52
• Maximum Entropy Exploration Without The Rollouts (2026)0.00
• Complexity-regularized Proximal Policy Optimization (2025)0.00
• Beyond Exact Gradients: Convergence Of Stochastic Soft-max Policy Gradient Methods With Entropy Regularization (2021)2.26