Between Rate-distortion Theory & Value Equivalence In Model-based Reinforcement Learning

Abstract

The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment. In this work, we entertain an extreme scenario wherein some combination of immense environment complexity and limited agent capacity entirely precludes identifying an exactly value-equivalent model. In light of this, we embrace a notion of approximate value equivalence and introduce an algorithm for incrementally

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyarumugam2022between

Related papers

• Deciding What To Model: Value-equivalent Sampling For Reinforcement Learning (2022)0.00
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• On Value Functions And The Agent-environment Boundary (2019)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• On The Limited Representational Power Of Value Functions And Its Links To Statistical (in)efficiency (2024)0.00
• Value-distributional Model-based Reinforcement Learning (2023)1.56
• A Distributional Perspective On Reinforcement Learning (2017)0.00
• Blending MPC & Value Function Approximation For Efficient Reinforcement Learning (2020)0.00