DIFFER: Decomposing Individual Reward For Fair Experience Replay In Multi-agent Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyhu2023differ

Related papers

• Distributional Reward Estimation For Effective Multi-agent Deep Reinforcement Learning (2022)0.00
• REMAX: Relational Representation For Multi-agent Exploration (2020)2.26
• Adaptive Value Decomposition With Greedy Marginal Contribution Computation For Cooperative Multi-agent Reinforcement Learning (2023)3.58
• Discovering Individual Rewards In Collective Behavior Through Inverse Multi-agent Reinforcement Learning (2023)0.00
• Locality Matters: A Scalable Value Decomposition Approach For Cooperative Multi-agent Reinforcement Learning (2021)0.00
• Revisiting Some Common Practices In Cooperative Multi-agent Reinforcement Learning (2022)0.00
• Inducing Cooperation Via Team Regret Minimization Based Multi-agent Deep Reinforcement Learning (2019)0.00
• Policy Distillation And Value Matching In Multiagent Reinforcement Learning (2019)10.48