Benchmarking Robustness And Generalization In Multi-agent Systems: A Case Study On Neural MMO

Abstract

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. The competition combines relatively complex environment design with large numbers of agents in the environment. The top submissions demonstrate strong success on this task using mostly standard reinforcement learning (RL) methods combined with domain-specific engineering. We summarize the competition design and results and suggest that, as an academic community, competitions may be a powerful approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2023benchmarking

Related papers

• The Neurips 2022 Neural MMO Challenge: A Massively Multiagent Competition With Specialization And Trade (2023)0.00
• Massively Multiagent Minigames For Training Generalist Agents (2024)0.00
• Neural MMO 2.0: A Massively Multi-task Addition To Massively Multi-agent Learning (2023)0.00
• Neural MMO V1.3: A Massively Multiagent Game Environment For Training And Evaluating Neural Networks (2020)0.00
• Smacv2: An Improved Benchmark For Cooperative Multi-agent Reinforcement Learning (2022)5.24
• Fightladder: A Benchmark For Competitive Multi-agent Reinforcement Learning (2024)0.00
• The Multi-agent Reinforcement Learning In Malm\"o (MARL\"O) Competition (2019)0.00
• Deep Multiagent Reinforcement Learning: Challenges And Directions (2021)0.00