Mitigating Off-policy Bias In Actor-critic Methods With One-step Q-learning: A Novel Correction Approach

Abstract

Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent's policy and collected data increases. Although the well-studied importance sampling and off-policy policy gradient techniques were proposed to compensate for this discrepancy, they usually require a collection of long trajectories and induce additional problems such as vanishing/exploding gradients or discarding many useful experiences, which eventually increases the computational complexity. Moreover, their generalization to either continuous action domains or policies approximated by deterministic deep neural networks is strictly limited. To overcome these limitations, we introduce a novel policy similarity measure to mitigate the effects of such discrepancy in continuous control. Our method offers an ad

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysaglam2022mitigating

Related papers

• Estimation Error Correction In Deep Reinforcement Learning For Deterministic Actor-critic Methods (2021)7.16
• Q-prop: Sample-efficient Policy Gradient With An Off-policy Critic (2016)0.00
• Neural Network Compatible Off-policy Natural Actor-critic Algorithm (2021)0.00
• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• Handling Cost And Constraints With Off-policy Deep Reinforcement Learning (2023)0.00
• Solving Continuous Control Via Q-learning (2022)0.00
• Sample-efficient Model-free Reinforcement Learning With Off-policy Critics (2019)9.60
• Local Advantage Actor-critic For Robust Multi-agent Deep Reinforcement Learning (2021)7.81