Greedy Based Value Representation For Optimal Coordination In Multi-agent Reinforcement Learning
2022 · Lipeng Wan, Zeyang Liu, Xingyu Chen, et al.
Abstract
Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning methods with linear value decomposition (LVD) or monotonic value decomposition (MVD) suffer from relative overgeneralization. As a result, they can not ensure optimal consistency (i.e., the correspondence between individual greedy actions and the maximal true Q value). In this paper, we derive the expression of the joint Q value function of LVD and MVD. According to the expression, we draw a transition diagram, where each self-transition node (STN) is a possible convergence. To ensure optimal consistency, the optimal node is required to be the unique STN. Therefore, we propose the greedy-based value representation (GVR), which turns the optimal node into an STN via inferior target shaping and further eliminates the non-optimal STNs via superior experience replay. In addition, GVR achieves an adaptive trade-off between optimality and stability. Our method outperforms state-of-the-art b
Authors
(none)
Tags
Stats
Related papers
- Greedy-based Value Representation For Optimal Coordination In Multi-agent Reinforcement Learning (2021)0.00
- Adaptive Value Decomposition With Greedy Marginal Contribution Computation For Cooperative Multi-agent Reinforcement Learning (2023)3.58
- Beyond Monotonicity: Revisiting Factorization Principles In Multi-agent Q-learning (2025)0.00
- Qatten: A General Framework For Cooperative Multiagent Reinforcement Learning (2020)0.00
- Understanding Value Decomposition Algorithms In Deep Cooperative Multi-agent Reinforcement Learning (2022)0.00
- Residual Q-networks For Value Function Factorizing In Multi-agent Reinforcement Learning (2022)10.21
- Towards Understanding Cooperative Multi-agent Q-learning With Value Factorization (2020)0.00
- An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00