What Should I Know? Using Meta-gradient Descent For Predictive Feature Discovery In A Single Stream Of Experience

Abstract

In computational reinforcement learning, a growing body of work seeks to construct an agent's perception of the world through predictions of future sensations; predictions about environment observations are used as additional input features to enable better goal-directed decision-making. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. This challenge is especially apparent in continual learning problems where a single stream of experience is available to a singular agent. As a primary contribution, we introduce a meta-gradient descent process by which an agent learns 1) what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward -- all during a single ongoing process of continual learning. In this manuscript we consider predictions expressed as General Value Functi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykearney2022what

Related papers

• Finding Useful Predictions By Meta-gradient Descent To Improve Decision-making (2021)0.00
• Meta-gradient Reinforcement Learning With An Objective Discovered Online (2020)0.00
• Meta-value Learning: A General Framework For Learning With Learning Awareness (2023)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Deep Online Learning Via Meta-learning: Continual Adaptation For Model-based RL (2018)0.00
• A Policy Gradient Algorithm For Learning To Learn In Multiagent Reinforcement Learning (2020)0.00
• One Step At A Time: Pros And Cons Of Multi-step Meta-gradient Reinforcement Learning (2021)0.00
• Understanding Self-predictive Learning For Reinforcement Learning (2022)0.00