Improving Fictitious Play Reinforcement Learning With Expanding Models

Abstract

Fictitious play with reinforcement learning is a general and effective framework for zero-sum games. However, using the current deep neural network models, the implementation of fictitious play faces crucial challenges. Neural network model training employs gradient descent approaches to update all connection weights, and thus is easy to forget the old opponents after training to beat the new opponents. Existing approaches often maintain a pool of historical policy models to avoid the forgetting. However, learning to beat a pool in stochastic games, i.e., a wide distribution over policy models, is either sample-consuming or insufficient to exploit all models with limited amount of samples. In this paper, we propose a learning process with neural fictitious play to alleviate the above issues. We train a single model as our policy model, which consists of sub-models and a selector. Everytime facing a new opponent, the model is expanded by adding a new sub-model, where only the new sub-mo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyqin2019improving

Related papers

• Anticipatory Fictitious Play (2022)0.00
• Fictitious Play In Zero-sum Stochastic Games (2020)0.00
• Provably Efficient Fictitious Play Policy Optimization For Zero-sum Markov Games With Structured Transitions (2022)0.00
• Deep Reinforcement Learning From Self-play In Imperfect-information Games (2016)0.00
• Mastering Strategy Card Game (legends Of Code And Magic) Via End-to-end Policy And Optimistic Smooth Fictitious Play (2023)0.00
• Learning Fast Changing Slow In Spiking Neural Networks (2024)3.58
• Offline Fictitious Self-play For Competitive Games (2024)0.00
• Transformer Based Reinforcement Learning For Games (2019)0.00