Task-agnostic Exploration Via Policy Gradient Of A Non-parametric State Entropy Estimate

Abstract

In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy? In this paper, we argue that the entropy of the state distribution induced by finite-horizon trajectories is a sensible target. Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a non-parametric, \(k\)-nearest neighbors estimate of the state distribution entropy. In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics. Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning a variety of meaningful reward-based tasks downstream.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymutti2020task

Related papers

• The Importance Of Non-markovianity In Maximum State Entropy Exploration (2022)0.00
• Maximum Entropy Exploration Without The Rollouts (2026)0.00
• Provably Efficient Maximum Entropy Exploration (2018)0.00
• Maximum-entropy Exploration With Future State-action Visitation Measures (2026)0.00
• Fast Rates For Maximum Entropy Exploration (2023)0.00
• K-means Maximum Entropy Exploration (2022)0.00
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• Accelerating Reinforcement Learning With Value-conditional State Entropy Exploration (2023)0.00