Acting Upon Imagination: When To Trust Imagined Trajectories In Model Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) aims to learn model(s) of the environment dynamics that can predict the outcome of its actions. Forward application of the model yields so called imagined trajectories (sequences of action, predicted state-reward) used to optimize the set of candidate actions that maximize expected reward. The outcome, an ideal imagined trajectory or plan, is imperfect and typically MBRL relies on model predictive control (MPC) to overcome this by continuously re-planning from scratch, incurring thus major computational cost and increasing complexity in tasks with longer receding horizon. We propose uncertainty estimation methods for online evaluation of imagined trajectories to assess whether further planned actions can be trusted to deliver acceptable reward. These methods include comparing the error after performing the last action with the standard expected error and using model uncertainty to assess the deviation from expected outcomes. Additionally, we in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyremonda2021acting

Related papers

• Plan To Predict: Learning An Uncertainty-foreseeing Model For Model-based Reinforcement Learning (2023)0.00
• Self-correcting Models For Model-based Reinforcement Learning (2016)0.00
• Trust The Model Where It Trusts Itself -- Model-based Actor-critic With Uncertainty-aware Rollout Adaption (2024)0.00
• Explaining Conditions For Reinforcement Learning Behaviors From Real And Imagined Data (2020)0.00
• Planning With Exploration: Addressing Dynamics Bottleneck In Model-based Reinforcement Learning (2020)0.00
• Model Imitation For Model-based Reinforcement Learning (2019)0.00
• Learning To Reweight Imaginary Transitions For Model-based Reinforcement Learning (2021)0.00
• Bridging Imagination And Reality For Model-based Deep Reinforcement Learning (2020)0.00