The Importance Of Non-markovianity In Maximum State Entropy Exploration

Abstract

In the maximum state entropy exploration framework, an agent interacts with a reward-free environment to learn a policy that maximizes the entropy of the expected state visitations it is inducing. Hazan et al. (2019) noted that the class of Markovian stochastic policies is sufficient for the maximum state entropy objective, and exploiting non-Markovianity is generally considered pointless in this setting. In this paper, we argue that non-Markovianity is instead paramount for maximum state entropy exploration in a finite-sample regime. Especially, we recast the objective to target the expected entropy of the induced state visitations in a single trial. Then, we show that the class of non-Markovian deterministic policies is sufficient for the introduced objective, while Markovian policies suffer non-zero regret in general. However, we prove that the problem of finding an optimal non-Markovian policy is NP-hard. Despite this negative result, we discuss avenues to address the problem in a

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymutti2022the

Related papers

• Task-agnostic Exploration Via Policy Gradient Of A Non-parametric State Entropy Estimate (2020)0.00
• Provably Efficient Maximum Entropy Exploration (2018)0.00
• Fast Rates For Maximum Entropy Exploration (2023)0.00
• Maximum-entropy Exploration With Future State-action Visitation Measures (2026)0.00
• Maximum Entropy Exploration Without The Rollouts (2026)0.00
• Accelerating Reinforcement Learning With Value-conditional State Entropy Exploration (2023)0.00
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• K-means Maximum Entropy Exploration (2022)0.00