Solving Common-payoff Games With Approximate Policy Iteration

Abstract

For artificially intelligent learning systems to have widespread applicability in real-world settings, it is important that they be able to operate decentrally. Unfortunately, decentralized control is difficult -- computing even an epsilon-optimal joint policy is a NEXP complete problem. Nevertheless, a recently rediscovered insight -- that a team of agents can coordinate via common knowledge -- has given rise to algorithms capable of finding optimal joint policies in small common-payoff games. The Bayesian action decoder (BAD) leverages this insight and deep reinforcement learning to scale to games as large as two-player Hanabi. However, the approximations it uses to do so prevent it from discovering optimal joint policies even in games small enough to brute force optimal solutions. This work proposes CAPI, a novel algorithm which, like BAD, combines common knowledge with deep reinforcement learning. However, unlike BAD, CAPI prioritizes the propensity to discover optimal joint polici

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keysokota2021solving

Related papers

• Bayesian Action Decoder For Deep Multi-agent Reinforcement Learning (2018)0.00
• Adaptive Approximate Policy Iteration (2020)0.00
• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• Dual Policy Iteration (2018)0.00
• Breaking The Curse Of Multiagency: Provably Efficient Decentralized Multi-agent RL With Function Approximation (2023)0.00
• Multi-agent Policy Optimization With Approximatively Synchronous Advantage Estimation (2020)0.00
• Generalized Beliefs For Cooperative AI (2022)0.00
• Simplified Action Decoder For Deep Multi-agent Reinforcement Learning (2019)4.03