Off-policy Policy Gradient With State Distribution Correction

Abstract

We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what would be the distribution of states under the learned policy. Here we build on recent progress for estimating the ratio of the state distributions under behavior and evaluation policies for policy evaluation, and present an off-policy policy gradient optimization technique that can account for this mismatch in distributions. We present an illustrative example of why this is important and a theoretical convergence guarantee for our approach. Empirically, we compare our method in simulations to several strong baselines which do not correct for this mismatch, significantly improving in the quality of the policy discovered.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2019off

Related papers

• Off-policy Policy Gradient Algorithms By Constraining The State Distribution Shift (2019)0.00
• Analysis Of On-policy Policy Gradient Methods Under The Distribution Mismatch (2025)0.00
• Correcting Discount-factor Mismatch In On-policy Policy Gradient Methods (2023)0.00
• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• A Temporal-difference Approach To Policy Gradient Estimation (2022)0.00
• Off-oab: Off-policy Policy Gradient Method With Optimal Action-dependent Baseline (2024)0.00
• Distributional Policy Optimization: An Alternative Approach For Continuous Control (2019)0.00
• A Study Of Policy Gradient On A Class Of Exactly Solvable Models (2020)0.00