Importance Mixing: Improving Sample Reuse In Evolutionary Policy Search Methods

Abstract

Deep neuroevolution, that is evolutionary policy search methods based on deep neural networks, have recently emerged as a competitor to deep reinforcement learning algorithms due to their better parallelization capabilities. However, these methods still suffer from a far worse sample efficiency. In this paper we investigate whether a mechanism known as "importance mixing" can significantly improve their sample efficiency. We provide a didactic presentation of importance mixing and we explain how it can be extended to reuse more samples. Then, from an empirical comparison based on a simple benchmark, we show that, though it actually provides better sample efficiency, it is still far from the sample efficiency of deep reinforcement learning, though it is more stable.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypourchot2018importance

Related papers

• Combining Evolution And Deep Reinforcement Learning For Policy Search: A Survey (2022)12.25
• CEM-RL: Combining Evolutionary And Gradient-based Methods For Policy Search (2018)0.00
• Sample Dropout: A Simple Yet Effective Variance Reduction Technique In Deep Policy Optimization (2023)0.00
• Improving Sample Efficiency In Evolutionary RL Using Off-policy Ranking (2022)0.00
• Generalized Policy Improvement Algorithms With Theoretically Supported Sample Reuse (2022)5.24
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00
• Neighborhood Mixup Experience Replay: Local Convex Interpolation For Improved Sample Efficiency In Continuous Control Tasks (2022)0.00
• Solving Deep Reinforcement Learning Tasks With Evolution Strategies And Linear Policy Networks (2024)0.00