Policy Optimization Finds Nash Equilibrium In Regularized General-sum LQ Games

Abstract

In this paper, we investigate the impact of introducing relative entropy regularization on the Nash Equilibria (NE) of General-Sum \(N\)-agent games, revealing the fact that the NE of such games conform to linear Gaussian policies. Moreover, it delineates sufficient conditions, contingent upon the adequacy of entropy regularization, for the uniqueness of the NE within the game. As Policy Optimization serves as a foundational approach for Reinforcement Learning (RL) techniques aimed at finding the NE, in this work we prove the linear convergence of a policy optimization algorithm which (subject to the adequacy of entropy regularization) is capable of provably attaining the NE. Furthermore, in scenarios where the entropy regularization proves insufficient, we present a \(\delta\)-augmentation technique, which facilitates the achievement of an \(\epsilon\)-NE within the game.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzaman2024policy

Related papers

• Linear Convergence Of Independent Natural Policy Gradient In Games With Entropy Regularization (2024)3.58
• Learning Nash Equilibria In Zero-sum Stochastic Games Via Entropy-regularized Policy Approximation (2020)0.00
• Policy Optimization Reinforcement Learning With Entropy Regularization (2019)0.00
• Fast Policy Extragradient Methods For Competitive Games With Entropy Regularization (2021)0.00
• Understanding The Impact Of Entropy On Policy Optimization (2018)0.00
• Entropy Regularization For Mean Field Games With Learning (2020)0.00
• Policy Optimization For Markov Games: Unified Framework And Faster Convergence (2022)0.00
• Optimal Scheduling Of Entropy Regulariser For Continuous-time Linear-quadratic Reinforcement Learning (2022)4.52