Selection-expansion: A Unifying Framework For Motion-planning And Diversity Search Algorithms

Abstract

Reinforcement learning agents need a reward signal to learn successful policies. When this signal is sparse or the corresponding gradient is deceptive, such agents need a dedicated mechanism to efficiently explore their search space without relying on the reward. Looking for a large diversity of behaviors or using Motion Planning (MP) algorithms are two options in this context. In this paper, we build on the common roots between these two options to investigate the properties of two diversity search algorithms, the Novelty Search and the Goal Exploration Process algorithms. These algorithms look for diversity in an outcome space or behavioral space which is generally hand-designed to represent what matters for a given task. The relation to MP algorithms reveals that the smoothness, or lack of smoothness of the mapping between the policy parameter space and the outcome space plays a key role in the search efficiency. In particular, we show empirically that, if the mapping is smooth enou

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychenu2021selection

Related papers

• Effective Diversity In Population Based Reinforcement Learning (2020)0.00
• Learning In Sparse Rewards Settings Through Quality-diversity Algorithms (2022)0.00
• Diverse Policies Converge In Reward-free Markov Decision Processe (2023)0.00
• Improving Exploration In Evolution Strategies For Deep Reinforcement Learning Via A Population Of Novelty-seeking Agents (2017)0.00
• Multi-objective Model-based Policy Search For Data-efficient Learning With Sparse Rewards (2018)0.00
• Phasic Diversity Optimization For Population-based Reinforcement Learning (2024)0.00
• Coordinated Exploration Via Intrinsic Rewards For Multi-agent Reinforcement Learning (2019)0.00
• DGPO: Discovering Multiple Strategies With Diversity-guided Policy Optimization (2022)2.26