An Intrinsically-motivated Approach For Learning Highly Exploring And Fast Mixing Policies

Abstract

What is a good exploration strategy for an agent that interacts with an environment in the absence of external rewards? Ideally, we would like to get a policy driving towards a uniform state-action visitation (highly exploring) in a minimum number of steps (fast mixing), in order to ease efficient learning of any goal-conditioned policy later on. Unfortunately, it is remarkably arduous to directly learn an optimal policy of this nature. In this paper, we propose a novel surrogate objective for learning highly exploring and fast mixing policies, which focuses on maximizing a lower bound to the entropy of the steady-state distribution induced by the policy. In particular, we introduce three novel lower bounds, that lead to as many optimization problems, that tradeoff the theoretical guarantees with computational complexity. Then, we present a model-based reinforcement learning algorithm, IDE\(^\{3\}\)AL, to learn an optimal policy according to the introduced objective. Finally, we provid

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keymutti2019an

Related papers

• Task-agnostic Exploration Via Policy Gradient Of A Non-parametric State Entropy Estimate (2020)0.00
• Never Give Up: Learning Directed Exploration Strategies (2020)0.00
• Maximum Entropy Exploration Without The Rollouts (2026)0.00
• Provably Efficient Maximum Entropy Exploration (2018)0.00
• Intrinsic Reward Policy Optimization For Sparse-reward Environments (2026)0.00
• Near-optimal Conservative Exploration In Reinforcement Learning Under Episode-wise Constraints (2023)0.00
• MULEX: Disentangling Exploitation From Exploration In Deep RL (2019)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00