Unifying Gradient Estimators For Meta-reinforcement Learning Via Off-policy Evaluation

Abstract

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2021unifying

Related papers

• An Empirical Analysis Of Measure-valued Derivatives For Policy Gradients (2021)0.00
• An Analysis Of Measure-valued Derivatives For Policy Gradients (2022)2.26
• Hybrid Value Estimation For Off-policy Evaluation And Offline Reinforcement Learning (2022)0.00
• Meta-value Learning: A General Framework For Learning With Learning Awareness (2023)0.00
• Debiasing Meta-gradient Reinforcement Learning By Learning The Outer Value Function (2022)0.00
• Interpolated Policy Gradient: Merging On-policy And Off-policy Gradient Estimation For Deep Reinforcement Learning (2017)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Loaded Dice: Trading Off Bias And Variance In Any-order Score Function Estimators For Reinforcement Learning (2019)0.00