The Lagrangian Method For Solving Constrained Markov Games

Abstract

We propose the concept of a Lagrangian game to solve constrained Markov games. Such games model scenarios where agents face cost constraints in addition to their individual rewards, that depend on both agent joint actions and the evolving environment state over time. Constrained Markov games form the formal mechanism behind safe multiagent reinforcement learning, providing a structured model for dynamic multiagent interactions in a multitude of settings, such as autonomous teams operating under local energy and time constraints, for example. We develop a primal-dual approach in which agents solve a Lagrangian game associated with the current Lagrange multiplier, simulate cost and reward trajectories over a fixed horizon, and update the multiplier using accrued experience. This update rule generates a new Lagrangian game, initiating the next iteration. Our key result consists in showing that the sequence of solutions to these Lagrangian games yields a nonstationary Nash solution for the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydas2025the

Related papers

• Provably Efficient Generalized Lagrangian Policy Optimization For Safe Multi-agent Reinforcement Learning (2023)0.00
• Independent Learning In Constrained Markov Potential Games (2024)0.00
• Actor-critic Algorithms For Constrained Multi-agent Reinforcement Learning (2019)0.00
• Extended Markov Games To Learn Multiple Tasks In Multi-agent Reinforcement Learning (2020)3.58
• Independent And Decentralized Learning In Markov Potential Games (2022)0.00
• Hardness Of Independent Learning And Sparse Equilibrium Computation In Markov Games (2023)0.00
• Efficiently Computing Nash Equilibria In Adversarial Team Markov Games (2022)0.00
• Policy Optimization For Markov Games: Unified Framework And Faster Convergence (2022)0.00