Learning To Reweight Imaginary Transitions For Model-based Reinforcement Learning

Abstract

Model-based reinforcement learning (RL) is more sample efficient than model-free RL by using imaginary trajectories generated by the learned dynamics model. When the model is inaccurate or biased, imaginary trajectories may be deleterious for training the action-value and policy functions. To alleviate such problem, this paper proposes to adaptively reweight the imaginary transitions, so as to reduce the negative effects of poorly generated trajectories. More specifically, we evaluate the effect of an imaginary transition by calculating the change of the loss computed on the real samples when we use the transition to train the action-value and policy functions. Based on this evaluation criterion, we construct the idea of reweighting each imaginary transition by a well-designed meta-gradient algorithm. Extensive experimental results demonstrate that our method outperforms state-of-the-art model-based and model-free RL algorithms on multiple tasks. Visualization of our changing weights f

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhuang2021learning

Related papers

• Learning Multimodal Transition Dynamics For Model-based Reinforcement Learning (2017)0.00
• Value Gradient Weighted Model-based Reinforcement Learning (2022)0.00
• Model Imitation For Model-based Reinforcement Learning (2019)0.00
• Acting Upon Imagination: When To Trust Imagined Trajectories In Model Based Reinforcement Learning (2021)0.00
• Harnessing Mixed Offline Reinforcement Learning Datasets Via Trajectory Weighting (2023)0.00
• Learning Expected Emphatic Traces For Deep RL (2021)5.24
• Bridging Imagination And Reality For Model-based Deep Reinforcement Learning (2020)0.00
• Algorithmic Framework For Model-based Deep Reinforcement Learning With Theoretical Guarantees (2018)0.00