Increasing Entropy To Boost Policy Gradient Performance On Personalization Tasks

Abstract

In this effort, we consider the impact of regularization on the diversity of actions taken by policies generated from reinforcement learning agents trained using a policy gradient. Policy gradient agents are prone to entropy collapse, which means certain actions are seldomly, if ever, selected. We augment the optimization objective function for the policy with terms constructed from various \(\varphi\)-divergences and Maximum Mean Discrepancy which encourages current policies to follow different state visitation and/or action choice distribution than previously computed policies. We provide numerical experiments using MNIST, CIFAR10, and Spotify datasets. The results demonstrate the advantage of diversity-promoting policy regularization and that its use on gradient-based approaches have significantly improved performance on a variety of personalization tasks. Furthermore, numerical evidence is given to show that policy regularization increases performance without losing accuracy.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keystarnes2023increasing

Related papers

• Policy Optimization Reinforcement Learning With Entropy Regularization (2019)0.00
• Understanding The Impact Of Entropy On Policy Optimization (2018)0.00
• Diversity-inducing Policy Gradient: Using Maximum Mean Discrepancy To Find A Set Of Diverse Policies (2019)8.35
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Marginalized State Distribution Entropy Regularization In Policy Optimization (2019)0.00
• Diversity Actor-critic: Sample-aware Entropy Regularization For Sample-efficient Exploration (2020)0.00
• An Entropy Regularization Free Mechanism For Policy-based Reinforcement Learning (2021)0.00
• Regularization Matters In Policy Optimization (2019)2.68