Unified Reinforcement Q-learning For Mean Field Game And Control Problems

Abstract

We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The *same* algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space where the agent not only provides an action to the environment but also a distribution of the state in order to take into account the mean field feature of the problem. Importantly, we assume that the agent can not observe the population's distribution and needs to estimate it in a model-free manner. The asymptotic MFG and MFC problems are also presented in continuous time and space, and compared with classical (non-asymptotic or stationary) MFG and MFC problems. They lead to explicit solutions in the linear-quadratic (LQ) case that are used as benchmarks for the results of our algorithm.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyangiuli2020unified

Related papers

• Analysis Of Multiscale Reinforcement Q-learning Algorithms For Mean Field Control Games (2024)0.00
• Deep Reinforcement Learning For Infinite Horizon Mean Field Problems In Continuous Spaces (2023)3.58
• Model-free Mean-field Reinforcement Learning: Mean-field MDP And Mean-field Q-learning (2019)0.00
• Unified Continuous-time Q-learning For Mean-field Game And Mean-field Control Problems (2024)0.00
• Reinforcement Learning In Non-stationary Discrete-time Linear-quadratic Mean-field Games (2020)10.07
• Efficient And Scalable Deep Reinforcement Learning For Mean Field Control Games (2024)0.00
• Global Convergence Of Policy Gradient For Linear-quadratic Mean-field Control/game In Continuous Time (2020)0.00
• Continuous-time Q-learning For Mean-field Control Problems (2023)0.00