Scalable Offline Reinforcement Learning For Mean Field Games

Abstract

Reinforcement learning algorithms for mean-field games offer a scalable framework for optimizing policies in large populations of interacting agents. Existing methods often depend on online interactions or access to system dynamics, limiting their practicality in real-world scenarios where such interactions are infeasible or difficult to model. In this paper, we present Offline Munchausen Mirror Descent (Off-MMD), a novel mean-field RL algorithm that approximates equilibrium policies in mean-field games using purely offline data. By leveraging iterative mirror descent and importance sampling techniques, Off-MMD estimates the mean-field distribution from static datasets without relying on simulation or environment dynamics. Additionally, we incorporate techniques from offline reinforcement learning to address common issues like Q-value overestimation, ensuring robust policy learning even with limited data coverage. Our algorithm scales to complex environments and demonstrates strong per

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybrunnbauer2024scalable

Related papers

• Population-aware Online Mirror Descent For Mean-field Games By Deep Reinforcement Learning (2024)0.00
• Efficient And Scalable Deep Reinforcement Learning For Mean Field Control Games (2024)0.00
• Offline Fictitious Self-play For Competitive Games (2024)0.00
• A Single Online Agent Can Efficiently Learn Mean Field Games (2024)0.00
• Oracle-free Reinforcement Learning In Mean-field Games Along A Single Sample Path (2022)0.00
• Deep Reinforcement Learning For Infinite Horizon Mean Field Problems In Continuous Spaces (2023)3.58
• Networked Communication For Mean-field Games With Function Approximation And Empirical Mean-field Estimation (2024)0.00
• Policy Mirror Ascent For Efficient And Independent Learning In Mean Field Games (2022)0.00