Efficient And Robust Reinforcement Learning With Uncertainty-based Value Expansion

Abstract

By integrating dynamics models into model-free reinforcement learning (RL) methods, model-based value expansion (MVE) algorithms have shown a significant advantage in sample efficiency as well as value estimation. However, these methods suffer from higher function approximation errors than model-free methods in stochastic environments due to a lack of modeling the environmental randomness. As a result, their performance lags behind the best model-free algorithms in some challenging scenarios. In this paper, we propose a novel Hybrid-RL method that builds on MVE, namely the Risk Averse Value Expansion (RAVE). With imaginative rollouts generated by an ensemble of probabilistic dynamics models, we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on a range of challenging environments show that by modeling the uncertainty completely, RAVE substantially enhances the robustness of previous model-based methods, and yields state-of-th

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhou2019efficient

Related papers

• Model-based Value Estimation For Efficient Model-free Reinforcement Learning (2018)0.00
• Sample-efficient Reinforcement Learning With Stochastic Ensemble Value Expansion (2018)0.00
• Diminishing Return Of Value Expansion Methods In Model-based Reinforcement Learning (2023)0.00
• Robust Risk-sensitive Reinforcement Learning With Conditional Value-at-risk (2024)5.84
• Extreme Risk Mitigation In Reinforcement Learning Using Extreme Value Theory (2023)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• Smart Exploration In Reinforcement Learning Using Bounded Uncertainty Models (2025)0.00
• VIREL: A Variational Inference Framework For Reinforcement Learning (2018)0.00