The Value Equivalence Principle For Model-based Reinforcement Learning

Abstract

Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources of model-based RL agents are better used to build models that are directly useful for value-based planning. As our main contribution, we introduce the principle of value equivalence: two models are value equivalent with respect to a set of functions and policies if they yield the same Bellman updates. We propose a formulation of the model learning problem based on the value equivalence principle and analyze how the set of feasible solutions is impacted by the choice of policies and functions. Specifically, we show that, as we augment the set of policies and functions considered, the class of va

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygrimm2020the

Related papers

• Between Rate-distortion Theory & Value Equivalence In Model-based Reinforcement Learning (2022)0.00
• Deciding What To Model: Value-equivalent Sampling For Reinforcement Learning (2022)0.00
• Is There Value In Reinforcement Learning? (2025)0.00
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• On Value Functions And The Agent-environment Boundary (2019)0.00
• A Look At Value-based Decision-time Vs. Background Planning Methods Across Different Settings (2022)0.00
• On The Limited Representational Power Of Value Functions And Its Links To Statistical (in)efficiency (2024)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00