Selectively Sharing Experiences Improves Multi-agent Reinforcement Learning

Abstract

We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the art multi-agent RL algorithms. Further, sharing only a small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants. A reference implementation of our algorithm is available at https://github.com/mgerstgrasser/super.

Stats

• citations1
• S2 citations—
• github stars14
• HF likes0
• heat score4.61
• arxiv keygerstgrasser2023selectively

Code

• mgerstgrasser/super—

Related papers

• Selective Experience Sharing In Reinforcement Learning Enhances Interference Management (2025)0.00
• Experience Sharing Between Cooperative Reinforcement Learning Agents (2019)0.00
• Scaling Multi-agent Reinforcement Learning With Selective Parameter Sharing (2021)0.00
• Shared Experience Actor-critic For Multi-agent Reinforcement Learning (2020)0.00
• Cautiously-optimistic Knowledge Sharing For Cooperative Multi-agent Reinforcement Learning (2023)5.84
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00
• Prioritized Guidance For Efficient Multi-agent Reinforcement Learning Exploration (2019)0.00
• Safe And Robust Experience Sharing For Deterministic Policy Gradient Algorithms (2022)0.00