Multi-agent Natural Actor-critic Reinforcement Learning Algorithms

Abstract

Multi-agent actor-critic algorithms are an important part of the Reinforcement Learning paradigm. We propose three fully decentralized multi-agent natural actor-critic (MAN) algorithms in this work. The objective is to collectively find a joint policy that maximizes the average long-term return of these agents. In the absence of a central controller and to preserve privacy, agents communicate some information to their neighbors via a time-varying communication network. We prove convergence of all the 3 MAN algorithms to a globally asymptotically stable set of the ODE corresponding to actor update; these use linear function approximations. We show that the Kullback-Leibler divergence between policies of successive iterates is proportional to the objective function's gradient. We observe that the minimum singular value of the Fisher information matrix is well within the reciprocal of the policy parameter dimension. Using this, we theoretically show that the optimal value of the determini

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keytrivedi2021multi

Related papers

• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00
• A Multi-agent Off-policy Actor-critic Algorithm For Distributed Reinforcement Learning (2019)11.39
• Actor-critic Algorithms For Constrained Multi-agent Reinforcement Learning (2019)0.00
• Contrasting Centralized And Decentralized Critics In Multi-agent Reinforcement Learning (2021)0.00
• Natural Actor-critic Converges Globally For Hierarchical Linear Quadratic Regulator (2019)0.00
• Distributed Off-policy Actor-critic Reinforcement Learning With Policy Consensus (2019)11.67
• F2A2: Flexible Fully-decentralized Approximate Actor-critic For Cooperative Multi-agent Reinforcement Learning (2020)0.00