An Off-policy Policy Gradient Theorem Using Emphatic Weightings

Abstract

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient. In off-policy learning, however, where the behaviour policy is not necessarily attempting to learn and follow the optimal policy for the given task, the existence of such a theorem has been elusive. In this work, we solve this open problem by providing the first off-policy policy gradient theorem. The key to the derivation is the use of \(emphatic\) \(weightings\). We develop a new actor-critic algorithm\(\unicode\{x2014\}\)called Actor Critic with Emphatic weightings (ACE)\(\unicode\{x2014\}\)that approximates the simplified gradients provided by the theorem. We demonstrate in a simple counterexample that previous off-policy policy gradient methods\(\unicode\{x2014\}\

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyimani2018an

Related papers

• Off-policy Actor-critic With Emphatic Weightings (2021)0.00
• Beyond The Policy Gradient Theorem For Efficient Policy Updates In Actor-critic Algorithms (2022)0.00
• Compatible Gradient Approximations For Actor-critic Algorithms (2024)0.00
• All-action Policy Gradient Methods: A Numerical Integration Approach (2019)0.00
• Convergent Actor-critic Algorithms Under Off-policy Training And Function Approximation (2018)0.00
• Policy Gradient Using Weak Derivatives For Reinforcement Learning (2020)0.00
• An Empirical Analysis Of Measure-valued Derivatives For Policy Gradients (2021)0.00
• An Approximate Policy Iteration Viewpoint Of Actor-critic Algorithms (2022)2.26