Convergence Of Actor-critic Learning For Mean Field Games And Mean Field Control In Continuous Spaces

Abstract

We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit soluti

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfouque2025convergence

Related papers

• Deep Reinforcement Learning For Infinite Horizon Mean Field Problems In Continuous Spaces (2023)3.58
• Analysis Of Multiscale Reinforcement Q-learning Algorithms For Mean Field Control Games (2024)0.00
• Efficient And Scalable Deep Reinforcement Learning For Mean Field Control Games (2024)0.00
• Learning Mean-field Games Through Mean-field Actor-critic Flow (2025)0.00
• Actor-critic Learning For Mean-field Control In Continuous Time (2023)0.00
• Actor Critic Learning Algorithms For Mean-field Control With Moment Neural Networks (2023)0.00
• Global Convergence Of Policy Gradient For Linear-quadratic Mean-field Control/game In Continuous Time (2020)0.00
• Unified Reinforcement Q-learning For Mean Field Game And Control Problems (2020)0.00