Deciding What To Model: Value-equivalent Sampling For Reinforcement Learning

Abstract

The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment, whose underlying dynamics likely exceed the agent's capacity for representation. In this work, we consider the scenario where agent limitations may entirely preclude identifying an exactly value-equivalent model, immediately giving rise to a trade-off between identifying a model that is simple enough to learn whil

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyarumugam2022deciding

Related papers

• Between Rate-distortion Theory & Value Equivalence In Model-based Reinforcement Learning (2022)0.00
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Model-based Value Estimation For Efficient Model-free Reinforcement Learning (2018)0.00
• On The Limited Representational Power Of Value Functions And Its Links To Statistical (in)efficiency (2024)0.00
• On The Model-misspecification In Reinforcement Learning (2023)0.00
• Online Model Selection For Reinforcement Learning With Function Approximation (2020)0.00
• Model-free Reinforcement Learning For Model-based Control: Towards Safe, Interpretable And Sample-efficient Agents (2025)0.00