Constrained Model-based Reinforcement Learning With Robust Cross-entropy Method

Abstract

This paper studies the constrained/safe reinforcement learning (RL) problem with sparse indicator signals for constraint violations. We propose a model-based approach to enable RL agents to effectively explore the environment with unknown system dynamics and environment constraints given a significantly small number of violation budgets. We employ the neural network ensemble model to estimate the prediction uncertainty and use model predictive control as the basic control framework. We propose the robust cross-entropy method to optimize the control sequence considering the model uncertainty and constraints. We evaluate our methods in the Safety Gym environment. The results show that our approach learns to complete the tasks with a much smaller number of constraint violations than state-of-the-art baselines. Additionally, we are able to achieve several orders of magnitude better sample efficiency when compared with constrained model-free RL approaches. The code is available at https://g

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2020constrained

Related papers

• Conservative And Adaptive Penalty For Model-based Safe Reinforcement Learning (2021)0.00
• Robust Model-free Reinforcement Learning With Multi-objective Bayesian Optimization (2019)11.08
• On The Robustness Of Safe Reinforcement Learning Under Observational Perturbations (2022)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Reinforcement Learning With Convex Constraints (2019)0.00
• Online Robust Reinforcement Learning With Model Uncertainty (2021)0.00
• Model-based Safe Deep Reinforcement Learning Via A Constrained Proximal Policy Optimization Algorithm (2022)5.24
• Robust Model-based Reinforcement Learning With An Adversarial Auxiliary Model (2024)0.00