Communication-efficient Policy Gradient Methods For Distributed Reinforcement Learning

Abstract

This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed l

Stats

• citations54
• S2 citations—
• github stars0
• HF likes0
• heat score13.05
• arxiv keychen2018communication

Related papers

• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00
• Asynchronous Policy Gradient Aggregation For Efficient Distributed Reinforcement Learning (2025)0.00
• Distributed Policy Gradient With Variance Reduction In Multi-agent Reinforcement Learning (2021)0.00
• Scalable Centralized Deep Multi-agent Reinforcement Learning Via Policy Gradients (2018)0.00
• The Gradient Convergence Bound Of Federated Multi-agent Reinforcement Learning With Efficient Communication (2021)0.00
• Descent-guided Policy Gradient For Scalable Cooperative Multi-agent Learning (2026)0.00
• A Policy Gradient Algorithm For Learning To Learn In Multiagent Reinforcement Learning (2020)0.00
• Fully Asynchronous Policy Evaluation In Distributed Reinforcement Learning Over Networks (2020)9.03