Mind The Model, Not The Agent: The Primacy Bias In Model-based RL

Abstract

The primacy bias in model-free reinforcement learning (MFRL), which refers to the agent's tendency to overfit early data and lose the ability to learn from new data, can significantly decrease the performance of MFRL algorithms. Previous studies have shown that employing simple techniques, such as resetting the agent's parameters, can substantially alleviate the primacy bias in MFRL. However, the primacy bias in model-based reinforcement learning (MBRL) remains unexplored. In this work, we focus on investigating the primacy bias in MBRL. We begin by observing that resetting the agent's parameters harms its performance in the context of MBRL. We further find that the primacy bias in MBRL is more closely related to the primacy bias of the world model instead of the primacy bias of the agent. Based on this finding, we propose \textit\{world model resetting\}, a simple yet effective technique to alleviate the primacy bias in MBRL. We apply our method to two different MBRL algorithms, MBPO

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyqiao2023mind

Related papers

• Reset-free Reinforcement Learning With World Models (2024)0.00
• How To Fine-tune The Model: Unified Model Shift And Model Bias Policy Optimization (2023)0.00
• Objective Mismatch In Model-based Reinforcement Learning (2020)0.00
• Self-correcting Models For Model-based Reinforcement Learning (2016)0.00
• When To Update Your Model: Constrained Model-based Reinforcement Learning (2022)2.26
• The Virtues Of Laziness In Model-based RL: A Unified Objective And Algorithms (2023)0.00
• An Analysis Of Model-based Reinforcement Learning From Abstracted Observations (2022)0.00
• Mitigating Planner Overfitting In Model-based Reinforcement Learning (2018)0.00