Distributed Off-policy Actor-critic Reinforcement Learning With Policy Consensus

Abstract

In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional consensus step to let all the agents asymptotically achieve agreement on the global optimal policy function. The convergence analysis of the proposed algorithm is provided and the effectiveness of the proposed algorithm is validated using a distributed resource allocation example. Compared to relevant distributed actor critic methods, here the agents do not share information about their local tasks, but instead they coordinate to estimate the global policy function.

Stats

• citations35
• S2 citations—
• github stars0
• HF likes0
• heat score11.67
• arxiv keyzhang2019distributed

Related papers

• A Multi-agent Off-policy Actor-critic Algorithm For Distributed Reinforcement Learning (2019)11.39
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Local Advantage Actor-critic For Robust Multi-agent Deep Reinforcement Learning (2021)7.81
• Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00
• Resilient Consensus-based Multi-agent Reinforcement Learning With Function Approximation (2021)0.00
• Multi-agent Natural Actor-critic Reinforcement Learning Algorithms (2021)3.58
• Communication-efficient Actor-critic Methods For Homogeneous Markov Games (2022)0.00
• Parameter Sharing Deep Deterministic Policy Gradient For Cooperative Multi-agent Reinforcement Learning (2017)0.00