Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning

Abstract

Two hitherto disconnected threads of research, diverse exploration (DE) and maximum entropy RL have addressed a wide range of problems facing reinforcement learning algorithms via ostensibly distinct mechanisms. In this work, we identify a connection between these two approaches. First, a discriminator-based diversity objective is put forward and connected to commonly used divergence measures. We then extend this objective to the maximum entropy framework and propose an algorithm Maximum Entropy Diverse Exploration (MEDE) which provides a principled method to learn diverse behaviors. A theoretical investigation shows that the set of policies learned by MEDE capture the same modalities as the optimal maximum entropy policy. In effect, the proposed algorithm disentangles the maximum entropy policy into its diverse, constituent policies. Experiments show that MEDE is superior to the state of the art in learning high performing and diverse policies.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycohen2019maximum

Related papers

• Maximum Entropy-regularized Multi-goal Reinforcement Learning (2019)0.00
• Fast Rates For Maximum Entropy Exploration (2023)0.00
• Diverse Policies Converge In Reward-free Markov Decision Processe (2023)0.00
• Diverse Exploration For Fast And Safe Policy Improvement (2018)4.52
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• MULEX: Disentangling Exploitation From Exploration In Deep RL (2019)0.00
• Maximum Entropy RL (provably) Solves Some Robust RL Problems (2021)0.00
• A Diffusion Model Framework For Maximum Entropy Reinforcement Learning (2025)0.00