T-soft Update Of Target Network For Deep Reinforcement Learning

Abstract

This paper proposes a new robust update rule of target network for deep reinforcement learning (DRL), to replace the conventional update rule, given as an exponential moving average. The target network is for smoothly generating the reference signals for a main network in DRL, thereby reducing learning variance. The problem with its conventional update rule is the fact that all the parameters are smoothly copied with the same speed from the main network, even when some of them are trying to update toward the wrong directions. This behavior increases the risk of generating the wrong reference signals. Although slowing down the overall update speed is a naive way to mitigate wrong updates, it would decrease learning speed. To robustly update the parameters while keeping learning speed, a t-soft update method, which is inspired by student-t distribution, is derived with reference to the analogy between the exponential moving average and the normal distribution. Through the analysis of the

Stats

• citations60
• S2 citations—
• github stars0
• HF likes0
• heat score13.39
• arxiv keykobayashi2020t

Related papers

• Breaking The Deadly Triad With A Target Network (2021)0.00
• Faster Deep Reinforcement Learning With Slower Online Network (2021)0.00
• MAD-TD: Model-augmented Data Stabilizes High Update Ratio RL (2024)0.00
• Mitigating Estimation Errors By Twin Td-regularized Actor And Critic For Deep Reinforcement Learning (2023)0.00
• Pseudo-quantized Actor-critic Algorithm For Robustness To Noisy Temporal Difference Error (2026)0.00
• Adam On Local Time: Addressing Nonstationarity In RL With Relative Adam Timesteps (2024)0.00
• Target-based Temporal Difference Learning (2019)0.00
• The Role Of Target Update Frequencies In Q-learning (2026)0.00