The Curse Of Diversity In Ensemble-based Exploration

Abstract

We uncover a surprising phenomenon in deep reinforcement learning: training a diverse ensemble of data-sharing agents -- a well-established exploration strategy -- can significantly impair the performance of the individual ensemble members when compared to standard single-agent training. Through careful analysis, we attribute the degradation in performance to the low proportion of self-generated data in the shared training data for each ensemble member, as well as the inefficiency of the individual ensemble members to learn from such highly off-policy data. We thus name this phenomenon the curse of diversity. We find that several intuitive solutions -- such as a larger replay buffer or a smaller ensemble size -- either fail to consistently mitigate the performance loss or undermine the advantages of ensembling. Finally, we demonstrate the potential of representation learning to counteract the curse of diversity with a novel method named Cross-Ensemble Representation Learning (CERL) in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylin2024the

Related papers

• Effective Diversity In Population Based Reinforcement Learning (2020)0.00
• The Impact Of Behavioral Diversity In Multi-agent Reinforcement Learning (2024)0.00
• SEERL: Sample Efficient Ensemble Reinforcement Learning (2020)2.26
• DEFT: Diverse Ensembles For Fast Transfer In Reinforcement Learning (2022)0.00
• Collaborative Training Of Heterogeneous Reinforcement Learning Agents In Environments With Sparse Rewards: What And When To Share? (2022)6.34
• How Exploration Breaks Cooperation In Shared-policy Multi-agent Reinforcement Learning (2026)0.00
• Collaborative Evolutionary Reinforcement Learning (2019)0.00
• Quantifying The Effects Of Environment And Population Diversity In Multi-agent Reinforcement Learning (2021)9.03