Improved Communication Efficiency In Federated Natural Policy Gradient Via Admm-based Gradient Updates

Abstract

Federated reinforcement learning (FedRL) enables agents to collaboratively train a global policy without sharing their individual data. However, high communication overhead remains a critical bottleneck, particularly for natural policy gradient (NPG) methods, which are second-order. To address this issue, we propose the FedNPG-ADMM framework, which leverages the alternating direction method of multipliers (ADMM) to approximate global NPG directions efficiently. We theoretically demonstrate that using ADMM-based gradient updates reduces communication complexity from \(\{O\}(\{d^\{2\}\})\) to \(\{O\}(\{d\})\) at each iteration, where \(d\) is the number of model parameters. Furthermore, we show that achieving an \(\epsilon\)-error stationary convergence requires \(\{O\}(\frac\{1\}\{(1-\gamma)^\{2\}\{\epsilon\}\})\) iterations for discount factor \(\gamma\), demonstrating that FedNPG-ADMM maintains the same convergence rate as the standard FedNPG. Through evaluation of the proposed algori

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylan2023improved

Related papers

• Asynchronous Federated Reinforcement Learning With Policy Gradient Updates: Algorithm Design And Convergence Analysis (2024)0.00
• Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
• Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00
• The Gradient Convergence Bound Of Federated Multi-agent Reinforcement Learning With Efficient Communication (2021)0.00
• Symmetric (optimistic) Natural Policy Gradient For Multi-agent Learning With Parameter Convergence (2022)0.00
• Communication-efficient Policy Gradient Methods For Distributed Reinforcement Learning (2018)13.05
• Descent-guided Policy Gradient For Scalable Cooperative Multi-agent Learning (2026)0.00
• Global Convergence Of Natural Policy Gradient With Hessian-aided Momentum Variance Reduction (2024)0.00