The Nature Of Temporal Difference Errors In Multi-step Distributional Reinforcement Learning

Abstract

We study the multi-step off-policy learning approach to distributional RL. Despite the apparent similarity between value-based RL and distributional RL, our study reveals intriguing and fundamental differences between the two cases in the multi-step setting. We identify a novel notion of path-dependent distributional TD error, which is indispensable for principled multi-step distributional RL. The distinction from the value-based case bears important implications on concepts such as backward-view algorithms. Our work provides the first theoretical guarantees on multi-step off-policy distributional RL algorithms, including results that apply to the small number of existing approaches to multi-step distributional RL. In addition, we derive a novel algorithm, Quantile Regression-Retrace, which leads to a deep RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN on the Atari-57 benchmark. Collectively, we shed light on how unique challenges in multi-step distributional RL

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2022the

Related papers

• A Differential Perspective On Distributional Reinforcement Learning (2025)0.00
• Statistical Efficiency Of Distributional Temporal Difference Learning And Freedman's Inequality In Hilbert Spaces (2024)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Nonlinear Distributional Gradient Temporal-difference Learning (2018)0.00
• The Statistical Benefits Of Quantile Temporal-difference Learning For Value Estimation (2023)0.00
• An Analysis Of Quantile Temporal-difference Learning (2023)0.00
• Distributional Reinforcement Learning For Multi-dimensional Reward Functions (2021)0.00
• A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76