Reward-free Policy Space Compression For Reinforcement Learning

Abstract

In reinforcement learning, we encode the potential behaviors of an agent interacting with an environment into an infinite set of policies, the policy space, typically represented by a family of parametric functions. Dealing with such a policy space is a hefty challenge, which often causes sample and computation inefficiencies. However, we argue that a limited number of policies are actually relevant when we also account for the structure of the environment and of the policy parameterization, as many of them would induce very similar interactions, i.e., state-action distributions. In this paper, we seek for a reward-free compression of the policy space into a finite set of representative policies, such that, given any policy \(\pi\), the minimum R\'enyi divergence between the state-action distributions of the representative policies and the state-action distribution of \(\pi\) is bounded. We show that this compression of the policy space can be formulated as a set cover problem, and it

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymutti2022reward

Related papers

• Discovering And Exploiting Sparse Rewards In A Learned Behavior Space (2021)0.00
• An Entropy Regularization Free Mechanism For Policy-based Reinforcement Learning (2021)0.00
• No Prior Mask: Eliminate Redundant Action For Deep Reinforcement Learning (2023)1.81
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Achieving Sample And Computational Efficient Reinforcement Learning By Action Space Reduction Via Grouping (2023)0.00
• Reward Constrained Policy Optimization (2018)0.00
• Discretizing Continuous Action Space With Unimodal Probability Distributions For On-policy Reinforcement Learning (2024)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00