Revisiting Parameter Sharing In Multi-agent Deep Reinforcement Learning

Abstract

Parameter sharing, where each agent independently learns a policy with fully shared parameters between all policies, is a popular baseline method for multi-agent deep reinforcement learning. Unfortunately, since all agents share the same policy network, they cannot learn different policies or tasks. This issue has been circumvented experimentally by adding an agent-specific indicator signal to observations, which we term "agent indication". Agent indication is limited, however, in that without modification it does not allow parameter sharing to be applied to environments where the action spaces and/or observation spaces are heterogeneous. This work formalizes the notion of agent indication and proves that it enables convergence to optimal policies for the first time. Next, we formally introduce methods to extend parameter sharing to learning in heterogeneous observation and action spaces, and prove that these methods allow for convergence to optimal policies. Finally, we experimentally

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyterry2020revisiting

Related papers

• Adaptive Parameter Sharing For Multi-agent Reinforcement Learning (2023)0.00
• Scaling Multi-agent Reinforcement Learning With Selective Parameter Sharing (2021)0.00
• Parameter Sharing Deep Deterministic Policy Gradient For Cooperative Multi-agent Reinforcement Learning (2017)0.00
• Parameter Sharing With Network Pruning For Scalable Multi-agent Deep Reinforcement Learning (2023)2.26
• Improving Global Parameter-sharing In Physically Heterogeneous Multi-agent Reinforcement Learning With Unified Action Space (2024)0.00
• Kaleidoscope: Learnable Masks For Heterogeneous Multi-agent Reinforcement Learning (2024)2.26
• How Exploration Breaks Cooperation In Shared-policy Multi-agent Reinforcement Learning (2026)0.00
• Hypermarl: Adaptive Hypernetworks For Multi-agent RL (2024)0.00