Gamma-nets: Generalizing Value Estimation Over Timescale

Abstract

We present \(\Gamma\)-nets, a method for generalizing value function estimation over timescale. By using the timescale as one of the estimator's inputs we can estimate value for arbitrary timescales. As a result, the prediction target for any timescale is available and we are free to train on multiple timescales at each timestep. Here we empirically evaluate \(\Gamma\)-nets in the policy evaluation setting. We first demonstrate the approach on a square wave and then on a robot arm using linear function approximation. Next, we consider the deep reinforcement learning setting using several Atari video games. Our results show that \(\Gamma\)-nets can be effective for predicting arbitrary timescales, with only a small cost in accuracy as compared to learning estimators for fixed timescales. \(\Gamma\)-nets provide a method for compactly making predictions at many timescales without requiring a priori knowledge of the task, making it a valuable contribution to ongoing work on model-based pl

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keysherstan2019gamma

Related papers

• Generative Temporal Difference Learning For Infinite-horizon Prediction (2020)0.00
• Adaptive Exploration For Data-efficient General Value Function Evaluations (2024)0.00
• Robust And Adaptive Temporal-difference Learning Using An Ensemble Of Gaussian Processes (2021)0.00
• Rethinking Value Function Learning For Generalization In Reinforcement Learning (2022)0.00
• Reducing Variance In Temporal-difference Value Estimation Via Ensemble Of Deep Networks (2022)0.00
• Target Networks And Over-parameterization Stabilize Off-policy Bootstrapping With Function Approximation (2024)0.00
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• Linear Function Approximation As A Computationally Efficient Method To Solve Classical Reinforcement Learning Challenges (2024)0.00