Finding Useful Predictions By Meta-gradient Descent To Improve Decision-making

Abstract

In computational reinforcement learning, a growing body of work seeks to express an agent's model of the world through predictions about future sensations. In this manuscript we focus on predictions expressed as General Value Functions: temporally extended estimates of the accumulation of a future signal. One challenge is determining from the infinitely many predictions that the agent could possibly make which might support decision-making. In this work, we contribute a meta-gradient descent method by which an agent can directly specify what predictions it learns, independent of designer instruction. To that end, we introduce a partially observable domain suited to this investigation. We then demonstrate that through interaction with the environment an agent can independently select predictions that resolve the partial-observability, resulting in performance similar to expertly chosen value functions. By learning, rather than manually specifying these predictions, we enable the agent t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykearney2021finding

Related papers

• What Should I Know? Using Meta-gradient Descent For Predictive Feature Discovery In A Single Stream Of Experience (2022)0.00
• Meta-value Learning: A General Framework For Learning With Learning Awareness (2023)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Meta-gradient Reinforcement Learning With An Objective Discovered Online (2020)0.00
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• Metacontrol For Adaptive Imagination-based Optimization (2017)0.00
• Reusable Options Through Gradient-based Meta Learning (2022)0.00
• Learning To Predict Without Looking Ahead: World Models Without Forward Prediction (2019)0.00