Planning With Exploration: Addressing Dynamics Bottleneck In Model-based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) is believed to have higher sample efficiency compared with model-free reinforcement learning (MFRL). However, MBRL is plagued by dynamics bottleneck dilemma. Dynamics bottleneck dilemma is the phenomenon that the performance of the algorithm falls into the local optimum instead of increasing when the interaction step with the environment increases, which means more data can not bring better performance. In this paper, we find that the trajectory reward estimation error is the main reason that causes dynamics bottleneck dilemma through theoretical analysis. We give an upper bound of the trajectory reward estimation error and point out that increasing the agent's exploration ability is the key to reduce trajectory reward estimation error, thereby alleviating dynamics bottleneck dilemma. Motivated by this, a model-based control method combined with exploration named MOdel-based Progressive Entropy-based Exploration (MOPE2) is proposed. We conduct

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2020planning

Related papers

• Objective Mismatch In Model-based Reinforcement Learning (2020)0.00
• Diminishing Return Of Value Expansion Methods In Model-based Reinforcement Learning (2023)0.00
• Plan To Predict: Learning An Uncertainty-foreseeing Model For Model-based Reinforcement Learning (2023)0.00
• Smart Exploration In Reinforcement Learning Using Bounded Uncertainty Models (2025)0.00
• Self-correcting Models For Model-based Reinforcement Learning (2016)0.00
• When To Update Your Model: Constrained Model-based Reinforcement Learning (2022)2.26
• Off-policy Reinforcement Learning With Model-based Exploration Augmentation (2025)0.00
• Acting Upon Imagination: When To Trust Imagined Trajectories In Model Based Reinforcement Learning (2021)0.00