K-means Maximum Entropy Exploration

Abstract

Exploration in high-dimensional, continuous spaces with sparse rewards is an open problem in reinforcement learning. Artificial curiosity algorithms address this by creating rewards that lead to exploration. Given a reinforcement learning algorithm capable of maximizing rewards, the problem reduces to finding an optimization objective consistent with exploration. Maximum entropy exploration uses the entropy of the state visitation distribution as such an objective. However, efficiently estimating the entropy of the state visitation distribution is challenging in high-dimensional, continuous spaces. We introduce an artificial curiosity algorithm based on lower bounding an approximation to the entropy of the state visitation distribution. The bound relies on a result we prove for non-parametric density estimation in arbitrary dimensions using k-means. We show that our approach is both computationally efficient and competitive on benchmarks for exploration in high-dimensional, continuous

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynedergaard2022k

Related papers

• Fast Rates For Maximum Entropy Exploration (2023)0.00
• Maximum Entropy Exploration Without The Rollouts (2026)0.00
• Maximum-entropy Exploration With Future State-action Visitation Measures (2026)0.00
• Task-agnostic Exploration Via Policy Gradient Of A Non-parametric State Entropy Estimate (2020)0.00
• Provably Efficient Maximum Entropy Exploration (2018)0.00
• R\'enyi State Entropy For Exploration Acceleration In Reinforcement Learning (2022)0.00
• The Importance Of Non-markovianity In Maximum State Entropy Exploration (2022)0.00
• Accelerating Reinforcement Learning With Value-conditional State Entropy Exploration (2023)0.00