How Much Can Change In A Year? Revisiting Evaluation In Multi-agent Reinforcement Learning

Abstract

Establishing sound experimental standards and rigour is important in any growing field of research. Deep Multi-Agent Reinforcement Learning (MARL) is one such nascent field. Although exciting progress has been made, MARL has recently come under scrutiny for replicability issues and a lack of standardised evaluation methodology, specifically in the cooperative setting. Although protocols have been proposed to help alleviate the issue, it remains important to actively monitor the health of the field. In this work, we extend the database of evaluation methodology previously published by containing meta-data on MARL publications from top-rated conferences and compare the findings extracted from this updated database to the trends identified in their work. Our analysis shows that many of the worrying trends in performance reporting remain. This includes the omission of uncertainty quantification, not reporting all relevant evaluation details and a narrowing of algorithmic development classe

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysingh2023how

Related papers

• Towards A Standardised Performance Evaluation Protocol For Cooperative MARL (2022)0.00
• Benchmarking Multi-agent Deep Reinforcement Learning Algorithms In Cooperative Tasks (2020)0.00
• A Review Of Cooperative Multi-agent Deep Reinforcement Learning (2019)19.08
• Adaptability In Multi-agent Reinforcement Learning: A Framework And Unified Review (2025)0.00
• Benchmarl: Benchmarking Multi-agent Reinforcement Learning (2023)5.58
• Empirical Study On Robustness And Resilience In Cooperative Multi-agent Reinforcement Learning (2025)0.00
• Dispelling The Mirage Of Progress In Offline MARL Through Standardised Baselines And Evaluation (2024)0.00
• Model-based Multi-agent Reinforcement Learning: Recent Progress And Prospects (2022)0.00