On Practical Robust Reinforcement Learning: Practical Uncertainty Set And Double-agent Algorithm

Abstract

Robust reinforcement learning (RRL) aims at seeking a robust policy to optimize the worst case performance over an uncertainty set of Markov decision processes (MDPs). This set contains some perturbed MDPs from a nominal MDP (N-MDP) that generate samples for training, which reflects some potential mismatches between training (i.e., N-MDP) and true environments. In this paper we present an elaborated uncertainty set by excluding some implausible MDPs from the existing sets. Under this uncertainty set, we develop a sample-based RRL algorithm (named ARQ-Learning) for tabular setting and characterize its finite-time error bound. Also, it is proved that ARQ-Learning converges as fast as the standard Q-Learning and robust Q-Learning while ensuring better robustness. We introduce an additional pessimistic agent which can tackle the major bottleneck for the extension of ARQ-Learning into the cases with larger or continuous state spaces. Incorporating this idea into RL algorithms, we propose do

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyhwang2023on

Related papers

• Online Robust Reinforcement Learning With Model Uncertainty (2021)0.00
• Sample-efficient Robust Multi-agent Reinforcement Learning In The Face Of Environmental Uncertainty (2024)0.00
• Combining Pessimism With Optimism For Robust And Efficient Model-based Deep Reinforcement Learning (2021)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• A Bayesian Approach To Robust Reinforcement Learning (2019)0.00
• Robust Model-based Reinforcement Learning With An Adversarial Auxiliary Model (2024)0.00
• Robust Multi-agent Reinforcement Learning With State Uncertainty (2023)0.00
• Robust Cooperative Multi-agent Reinforcement Learning:a Mean-field Type Game Perspective (2024)0.00