MDPGT: Momentum-based Decentralized Policy Gradient Tracking

Abstract

We propose a novel policy gradient method for multi-agent reinforcement learning, which leverages two different variance-reduction techniques and does not require large batches over iterations. Specifically, we propose a momentum-based decentralized policy gradient tracking (MDPGT) where a new momentum-based variance reduction technique is used to approximate the local policy gradient surrogate with importance sampling, and an intermediate parameter is adopted to track two consecutive policy gradient surrogates. Moreover, MDPGT provably achieves the best available sample complexity of \(\mathcal\{O\}(N^\{-1\}\epsilon^\{-3\})\) for converging to an \(\epsilon\)-stationary point of the global average of \(N\) local performance functions (possibly nonconcave). This outperforms the state-of-the-art sample complexity in decentralized model-free reinforcement learning, and when initialized with a single trajectory, the sample complexity matches those obtained by the existing decentralized po

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjiang2021mdpgt

Related papers

• Distributed Policy Gradient With Variance Reduction In Multi-agent Reinforcement Learning (2021)0.00
• Descent-guided Policy Gradient For Scalable Cooperative Multi-agent Learning (2026)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• Bregman Gradient Policy Optimization (2021)0.00
• Independent Policy Gradient For Large-scale Markov Potential Games: Sharper Rates, Function Approximation, And Game-agnostic Convergence (2022)0.00
• Global Convergence Of Natural Policy Gradient With Hessian-aided Momentum Variance Reduction (2024)0.00
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Symmetric (optimistic) Natural Policy Gradient For Multi-agent Learning With Parameter Convergence (2022)0.00