A Policy Gradient Algorithm For Learning To Learn In Multiagent Reinforcement Learning

Abstract

A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other simultaneously learning agents. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural non-stationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings. This is achieved by modeling our gradient updates to consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment. We show that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently comprises all key aspects of previous state of the art approaches on this topic

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykim2020a

Related papers

• Multi-agent Cooperation Through Learning-aware Policy Gradients (2024)0.00
• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00
• Scalable Centralized Deep Multi-agent Reinforcement Learning Via Policy Gradients (2018)0.00
• Non-stationary Policy Learning For Multi-timescale Multi-agent Reinforcement Learning (2023)5.24
• Policy Gradient With Self-attention For Model-free Distributed Nonlinear Multi-agent Games (2025)0.00
• Dimension-free Rates For Natural Policy Gradient In Multi-agent Reinforcement Learning (2021)0.00
• Descent-guided Policy Gradient For Scalable Cooperative Multi-agent Learning (2026)0.00
• Meta-value Learning: A General Framework For Learning With Learning Awareness (2023)0.00