Off-policy Reinforcement Learning With Loss Function Weighted By Temporal Difference Error

Abstract

Training agents via off-policy deep reinforcement learning (RL) requires a large memory, named replay memory, that stores past experiences used for learning. These experiences are sampled, uniformly or non-uniformly, to create the batches used for training. When calculating the loss function, off-policy algorithms assume that all samples are of the same importance. In this paper, we hypothesize that training can be enhanced by assigning different importance for each experience based on their temporal-difference (TD) error directly in the training objective. We propose a novel method that introduces a weighting factor for each experience when calculating the loss function at the learning stage. In addition to improving convergence speed when used with uniform sampling, the method can be combined with prioritization methods for non-uniform sampling. Combining the proposed method with prioritization methods improves sampling efficiency while increasing the performance of TD-based off-poli

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keypark2022off

Related papers

• Learning Expected Emphatic Traces For Deep RL (2021)5.24
• Discerning Temporal Difference Learning (2023)0.00
• Simplifying Deep Temporal Difference Learning (2024)0.00
• Preferential Temporal Difference Learning (2021)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• Finite-time Analysis Of Temporal Difference Learning With Experience Replay (2023)0.00
• Improving The Efficiency Of Off-policy Reinforcement Learning By Accounting For Past Decisions (2021)0.00
• Adaptive Temporal-difference Learning For Policy Evaluation With Per-state Uncertainty Estimates (2019)0.00