MESA: Cooperative Meta-exploration In Multi-agent Learning Through Exploiting State-action Space Structure

Abstract

Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-reward settings, caused by the larger variance exhibited in policy learning. This paper introduces MESA, a novel meta-exploration method for cooperative multi-agent learning. It learns to explore by first identifying the agents' high-rewarding joint state-action subspace from training tasks and then learning a set of diverse exploration policies to "cover" the subspace. These trained exploration policies can be integrated with any off-policy MARL algorithm for test-time tasks. We first showcase MESA's advantage in a multi-step matrix game. Furthermore, experiments show that with learned exploration policies, MESA achieves significantly better performance in sparse-reward tasks in several multi-agent particle environments and multi-agent MuJoCo environments, and exhibits t

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyzhang2024mesa

Related papers

• Graph Exploration For Effective Multi-agent Q-learning (2023)5.24
• REMAX: Relational Representation For Multi-agent Exploration (2020)2.26
• Exploiting Semantic Epsilon Greedy Exploration Strategy In Multi-agent Reinforcement Learning (2022)0.00
• Coordinated Exploration Via Intrinsic Rewards For Multi-agent Reinforcement Learning (2019)0.00
• Settling Decentralized Multi-agent Coordinated Exploration By Novelty Sharing (2024)4.52
• Ensemble Value Functions For Efficient Exploration In Multi-agent Reinforcement Learning (2023)0.00
• Maximum Entropy Heterogeneous-agent Reinforcement Learning (2023)0.00
• Prioritized Guidance For Efficient Multi-agent Reinforcement Learning Exploration (2019)0.00