Independent Policy Mirror Descent For Markov Potential Games: Scaling To Large Number Of Players

Abstract

Markov Potential Games (MPGs) form an important sub-class of Markov games, which are a common framework to model multi-agent reinforcement learning problems. In particular, MPGs include as a special case the identical-interest setting where all the agents share the same reward function. Scaling the performance of Nash equilibrium learning algorithms to a large number of agents is crucial for multi-agent systems. To address this important challenge, we focus on the independent learning setting where agents can only have access to their local information to update their own policy. In prior work on MPGs, the iteration complexity for obtaining \(\epsilon\)-Nash regret scales linearly with the number of agents \(N\). In this work, we investigate the iteration complexity of an independent policy mirror descent (PMD) algorithm for MPGs. We show that PMD with KL regularization, also known as natural policy gradient, enjoys a better \(\sqrt\{N\}\) dependence on the number of agents, improving

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyalatur2024independent

Related papers

• Independent Policy Gradient For Large-scale Markov Potential Games: Sharper Rates, Function Approximation, And Game-agnostic Convergence (2022)0.00
• Independent Natural Policy Gradient Always Converges In Markov Potential Games (2021)0.00
• Provably Fast Convergence Of Independent Natural Policy Gradient For Markov Potential Games (2023)0.00
• Policy Mirror Ascent For Efficient And Independent Learning In Mean Field Games (2022)0.00
• Independent Learning In Constrained Markov Potential Games (2024)0.00
• Independent And Decentralized Learning In Markov Potential Games (2022)0.00
• Learning Mirror Maps In Policy Mirror Descent (2024)0.00
• Empirical Policy Optimization For \(n\)-player Markov Games (2021)0.00