Population-aware Online Mirror Descent For Mean-field Games By Deep Reinforcement Learning

Abstract

Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems, but learning Nash equilibria in MFGs remains a challenging task. In this paper, we propose a deep reinforcement learning (DRL) algorithm that achieves population-dependent Nash equilibrium without the need for averaging or sampling from history, inspired by Munchausen RL and Online Mirror Descent. Through the design of an additional inner-loop replay buffer, the agents can effectively learn to achieve Nash equilibrium from any distribution, mitigating catastrophic forgetting. The resulting policy can be applied to various initial distributions. Numerical experiments on four canonical examples demonstrate our algorithm has better convergence properties than SOTA algorithms, in particular a DRL version of Fictitious Play for population-dependent policies.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2024population

Related papers

• Scalable Offline Reinforcement Learning For Mean Field Games (2024)0.00
• A Single Online Agent Can Efficiently Learn Mean Field Games (2024)0.00
• Policy Mirror Ascent For Efficient And Independent Learning In Mean Field Games (2022)0.00
• Efficient And Scalable Deep Reinforcement Learning For Mean Field Control Games (2024)0.00
• Generalization In Mean Field Games By Learning Master Policies (2021)7.81
• Deep Reinforcement Learning For Infinite Horizon Mean Field Problems In Continuous Spaces (2023)3.58
• Oracle-free Reinforcement Learning In Mean-field Games Along A Single Sample Path (2022)0.00
• Learning In Mean Field Games: A Survey (2022)0.00