Linear Convergence Of Independent Natural Policy Gradient In Games With Entropy Regularization

Abstract

This work focuses on the entropy-regularized independent natural policy gradient (NPG) algorithm in multi-agent reinforcement learning. In this work, agents are assumed to have access to an oracle with exact policy evaluation and seek to maximize their respective independent rewards. Each individual's reward is assumed to depend on the actions of all the agents in the multi-agent system, leading to a game between agents. We assume all agents make decisions under a policy with bounded rationality, which is enforced by the introduction of entropy regularization. In practice, a smaller regularization implies the agents are more rational and behave closer to Nash policies. On the other hand, agents with larger regularization acts more randomly, which ensures more exploration. We show that, under sufficient entropy regularization, the dynamics of this system converge at a linear rate to the quantal response equilibrium (QRE). Although regularization assumptions prevent the QRE from approxim

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keysun2024linear

Related papers

• Fast Global Convergence Of Natural Policy Gradient Methods With Entropy Regularization (2020)0.00
• Linear Convergence Of Entropy-regularized Natural Policy Gradient With Linear Function Approximation (2021)6.34
• Provably Fast Convergence Of Independent Natural Policy Gradient For Markov Potential Games (2023)0.00
• Symmetric (optimistic) Natural Policy Gradient For Multi-agent Learning With Parameter Convergence (2022)0.00
• Policy Optimization Finds Nash Equilibrium In Regularized General-sum LQ Games (2024)0.00
• Matryoshka Policy Gradient For Entropy-regularized RL: Convergence And Global Optimality (2023)0.00
• Fast Policy Extragradient Methods For Competitive Games With Entropy Regularization (2021)0.00
• Independent Policy Gradient For Large-scale Markov Potential Games: Sharper Rates, Function Approximation, And Game-agnostic Convergence (2022)0.00