The Value-improvement Path: Towards Better Representations For Reinforcement Learning

Abstract

In value-based reinforcement learning (RL), unlike in supervised learning, the agent faces not a single, stationary, approximation problem, but a sequence of value prediction problems. Each time the policy improves, the nature of the problem changes, shifting both the distribution of states and their values. In this paper we take a novel perspective, arguing that the value prediction problems faced by an RL agent should not be addressed in isolation, but rather as a single, holistic, prediction problem. An RL algorithm generates a sequence of policies that, at least approximately, improve towards the optimal policy. We explicitly characterize the associated sequence of value functions and call it the value-improvement path. Our main idea is to approximate the value-improvement path holistically, rather than to solely track the value function of the current policy. Specifically, we discuss the impact that this holistic view of RL has on representation learning. We demonstrate that a rep

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keydabney2020the

Related papers

• Is There Value In Reinforcement Learning? (2025)0.00
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• Rethinking Model-based, Policy-based, And Value-based Reinforcement Learning Via The Lens Of Representation Complexity (2023)2.26
• Value-consistent Representation Learning For Data-efficient Reinforcement Learning (2022)0.00
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• General Policy Evaluation And Improvement By Learning To Identify Few But Crucial States (2022)0.00
• Disentangling Dynamics And Returns: Value Function Decomposition With Future Prediction (2019)0.00
• Kalman Meets Bellman: Improving Policy Evaluation Through Value Tracking (2020)0.00