Disentangling Dynamics And Returns: Value Function Decomposition With Future Prediction

Abstract

Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from the perspective of future prediction, through decomposing the value function into a reward-independent future dynamics part and a policy-independent trajectory return part. We then derive a practical deep RL algorithm from the above decomposition, consisting of a convolutional trajectory representation model, a conditional variational dynamics model to predict the expected representation of future trajectory and a convex trajectory return model that maps a trajectory representation to its return. Our algorithm is evaluated in MuJoCo continuous control tasks and shows superior results under both common settings and delayed reward settings.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2019disentangling

Related papers

• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• Pretrain Value, Not Reward: Decoupled Value Policy Optimization (2025)0.00
• Explainable Reinforcement Learning Via Temporal Policy Decomposition (2025)0.00
• Value Function Decomposition For Iterative Design Of Reinforcement Learning Agents (2022)0.00
• Hindsight Value Function For Variance Reduction In Stochastic Dynamic Environment (2021)2.26
• Beyond Exponentially Discounted Sum: Automatic Learning Of Return Function (2019)0.00
• Approximating Two Value Functions Instead Of One: Towards Characterizing A New Family Of Deep Reinforcement Learning Algorithms (2019)0.00