Model Based Meta Learning Of Critics For Policy Gradients

Abstract

Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in reinforcement learning. In this paper we present a framework to meta-learn the critic for gradient-based policy learning. Concretely, we propose a model-based bi-level optimization algorithm that updates the critics parameters such that the policy that is learned with the updated critic gets closer to solving the meta-training tasks. We illustrate that our algorithm leads to learned critics that resemble the ground truth Q function for a given task. Finally, after meta-training, the learned critic can be used to learn new policies for new unseen task and environment settings via model-free policy gradient optimization, without requiring a model. We present results that show the generalization capabilities of our learned critic to new tasks and dynamics when used to learn

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybechtle2022model

Related papers

• How To Learn A Useful Critic? Model-based Action-gradient-estimator Policy Optimization (2020)0.00
• Double Meta-learning For Data Efficient Policy Optimization In Non-stationary Environments (2020)0.00
• Model-based Adversarial Meta-reinforcement Learning (2020)0.00
• Meta-reinforcement Learning With Universal Policy Adaptation: Provable Near-optimality Under All-task Optimum Comparator (2024)0.00
• Stepsize Learning For Policy Gradient Methods In Contextual Markov Decision Processes (2023)2.26
• Learning To Explore With Meta-policy Gradient (2018)0.00
• Evolving Pareto-optimal Actor-critic Algorithms For Generalizability And Stability (2022)0.00
• Unifying Gradient Estimators For Meta-reinforcement Learning Via Off-policy Evaluation (2021)0.00