Pipeline PSRO: A Scalable Approach For Finding Approximate Nash Equilibria In Large Games

Abstract

Finding approximate Nash equilibria in zero-sum imperfect-information games is challenging when the number of information states is large. Policy Space Response Oracles (PSRO) is a deep reinforcement learning algorithm grounded in game theory that is guaranteed to converge to an approximate Nash equilibrium. However, PSRO requires training a reinforcement learning policy at each iteration, making it too slow for large games. We show through counterexamples and experiments that DCH and Rectified PSRO, two existing approaches to scaling up PSRO, fail to converge even in small games. We introduce Pipeline PSRO (P2SRO), the first scalable general method for finding approximate Nash equilibria in large zero-sum imperfect-information games. P2SRO is able to parallelize PSRO with convergence guarantees by maintaining a hierarchical pipeline of reinforcement learning workers, each training against the policies generated by lower levels in the hierarchy. We show that unlike existing methods, P2

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymcaleer2020pipeline

Related papers

• Simulation-free PSRO: Removing Game Simulation From Policy Space Response Oracles (2025)0.00
• A Generalized Training Approach For Multiagent Learning (2019)0.00
• Learning Equilibria In Mean-field Games: Introducing Mean-field PSRO (2021)0.00
• Fusion-psro: Nash Policy Fusion For Policy Space Response Oracles (2024)3.58
• Fictitious Cross-play: Learning Global Nash Equilibrium In Mixed Cooperative-competitive Games (2023)3.58
• A Unified Perspective On Deep Equilibrium Finding (2022)0.00
• On The Convergence Of Policy Gradient Methods To Nash Equilibria In General Stochastic Games (2022)0.00
• Independent Policy Gradient For Large-scale Markov Potential Games: Sharper Rates, Function Approximation, And Game-agnostic Convergence (2022)0.00