Oracles & Followers: Stackelberg Equilibria In Deep Multi-agent Reinforcement Learning

Abstract

Stackelberg equilibria arise naturally in a range of popular learning problems, such as in security games or indirect mechanism design, and have received increasing attention in the reinforcement learning literature. We present a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem, allowing a wide range of algorithmic design choices. We discuss how previous approaches can be seen as specific instantiations of this framework. As a key insight, we note that the design space allows for approaches not previously seen in the literature, for instance by leveraging multitask and meta-RL techniques for follower convergence. We propose one such approach using contextual policies, and evaluate it experimentally on both standard and novel benchmark domains, showing greatly improved sample efficiency compared to previous approaches. Finally, we explore the effect of adopting algorithm designs outside the borders of our framework.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygerstgrasser2022oracles

Related papers

• Can Reinforcement Learning Find Stackelberg-nash Equilibria In General-sum Markov Games With Myopic Followers? (2021)0.00
• Model-free Reinforcement Learning For Stochastic Stackelberg Security Games (2020)5.24
• Sample-efficient Learning Of Stackelberg Equilibria In General-sum Games (2021)0.00
• Inducing Stackelberg Equilibrium Through Spatio-temporal Sequential Decision-making In Multi-agent Reinforcement Learning (2023)7.50
• Actions Speak What You Want: Provably Sample-efficient Reinforcement Learning Of The Quantal Stackelberg Equilibrium From Strategic Feedbacks (2023)0.00
• Stackelberg Games For Learning Emergent Behaviors During Competitive Autocurricula (2023)5.84
• Algorithms In Multi-agent Systems: A Holistic Perspective From Reinforcement Learning And Game Theory (2020)0.00
• A Generalized Training Approach For Multiagent Learning (2019)0.00