Combining Pessimism With Optimism For Robust And Efficient Model-based Deep Reinforcement Learning

Abstract

In real-world tasks, reinforcement learning (RL) agents frequently encounter situations that are not present during training time. To ensure reliable performance, the RL agents need to exhibit robustness against worst-case situations. The robust RL framework addresses this challenge via a worst-case optimization between an agent and an adversary. Previous robust RL algorithms are either sample inefficient, lack robustness guarantees, or do not scale to large problems. We propose the Robust Hallucinated Upper-Confidence RL (RH-UCRL) algorithm to provably solve this problem while attaining near-optimal sample complexity guarantees. RH-UCRL is a model-based reinforcement learning (MBRL) algorithm that effectively distinguishes between epistemic and aleatoric uncertainty and efficiently explores both the agent and adversary decision spaces during policy learning. We scale RH-UCRL to complex tasks via neural networks ensemble models as well as neural network policies. Experimentally, we dem

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycuri2021combining

Related papers

• Robust Model-based Reinforcement Learning With An Adversarial Auxiliary Model (2024)0.00
• Efficient Model-based Reinforcement Learning Through Optimistic Policy Search And Planning (2020)0.00
• On Practical Robust Reinforcement Learning: Practical Uncertainty Set And Double-agent Algorithm (2023)3.58
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Online Robust Reinforcement Learning With Model Uncertainty (2021)0.00
• Efficient Model-based Multi-agent Reinforcement Learning Via Optimistic Equilibrium Computation (2022)0.00
• Efficient Adversarial Training Without Attacking: Worst-case-aware Robust Reinforcement Learning (2022)0.00
• Robust Model-free Reinforcement Learning With Multi-objective Bayesian Optimization (2019)11.08