Maximum Causal Entropy IRL In Mean-field Games And GNEP Framework For Forward RL

Abstract

This paper explores the use of Maximum Causal Entropy Inverse Reinforcement Learning (IRL) within the context of discrete-time stationary Mean-Field Games (MFGs) characterized by finite state spaces and an infinite-horizon, discounted-reward setting. Although the resulting optimization problem is non-convex with respect to policies, we reformulate it as a convex optimization problem in terms of state-action occupation measures by leveraging the linear programming framework of Markov Decision Processes. Based on this convex reformulation, we introduce a gradient descent algorithm with a guaranteed convergence rate to efficiently compute the optimal solution. Moreover, we develop a new method that conceptualizes the MFG problem as a Generalized Nash Equilibrium Problem (GNEP), enabling effective computation of the mean-field equilibrium for forward reinforcement learning (RL) problems and marking an advancement in MFG solution techniques. We further illustrate the practical applicability

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyanahtarci2024maximum

Related papers

• Kernel Based Maximum Entropy Inverse Reinforcement Learning For Mean-field Games (2025)0.00
• Adversarial Inverse Reinforcement Learning For Mean Field Games (2021)0.00
• Approximately Solving Mean Field Games Via Entropy-regularized Deep Reinforcement Learning (2021)0.00
• Inverse Reinforcement Learning With Explicit Policy Estimates (2021)2.26
• Deep Reinforcement Learning For Infinite Horizon Mean Field Problems In Continuous Spaces (2023)3.58
• Reinforcement Learning In Non-stationary Discrete-time Linear-quadratic Mean-field Games (2020)10.07
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• Model-based RL For Mean-field Games Is Not Statistically Harder Than Single-agent RL (2024)0.00