Robust Reinforcement Learning On State Observations With Learned Optimal Adversary

Abstract

We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2021robust

Related papers

• Robust Deep Reinforcement Learning Against Adversarial Perturbations On State Observations (2020)0.00
• Adversary Agnostic Robust Deep Reinforcement Learning (2020)6.77
• Efficient Adversarial Training Without Attacking: Worst-case-aware Robust Reinforcement Learning (2022)0.00
• On The Robustness Of Safe Reinforcement Learning Under Observational Perturbations (2022)0.00
• Belief-enriched Pessimistic Q-learning Against Adversarial State Perturbations (2024)1.56
• Robust Deep Reinforcement Learning With Adaptive Adversarial Perturbations In Action Space (2024)6.20
• Towards Robust Policy: Enhancing Offline Reinforcement Learning With Adversarial Attacks And Defenses (2024)3.58
• Regret-based Defense In Adversarial Reinforcement Learning (2023)0.00