Robust Adversarial Reinforcement Learning Via Bounded Rationality Curricula

Abstract

Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against destabilizing forces exercised by an adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high-dimensional control. In this paper, we propose a novel approach for adversarial RL based on entropy regularization to ease the complexity of the saddle point optimization problem. We show that the solution of this entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE), a generalization of Nash equilibria that accounts for bounded rationality, i.e., agents sometimes play random actions instead of optimal ones. Crucially, the connection between

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyreddi2023robust

Related papers

• Feasible Adversarial Robust Reinforcement Learning For Underspecified Environments (2022)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Robust Reinforcement Learning On State Observations With Learned Optimal Adversary (2021)0.00
• Robust Adversarial Policy Optimization Under Dynamics Uncertainty (2026)0.00
• Robust Model-based Reinforcement Learning With An Adversarial Auxiliary Model (2024)0.00
• Maximum Entropy RL (provably) Solves Some Robust RL Problems (2021)0.00
• On Practical Robust Reinforcement Learning: Practical Uncertainty Set And Double-agent Algorithm (2023)3.58
• Strategically Robust Multi-agent Reinforcement Learning With Linear Function Approximation (2026)0.00