An Experimental Comparison Between Temporal Difference And Residual Gradient With Neural Network Approximation

Abstract

Gradient descent or its variants are popular in training neural networks. However, in deep Q-learning with neural network approximation, a type of reinforcement learning, gradient descent (also known as Residual Gradient (RG)) is barely used to solve Bellman residual minimization problem. On the contrary, Temporal Difference (TD), an incomplete gradient descent method prevails. In this work, we perform extensive experiments to show that TD outperforms RG, that is, when the training leads to a small Bellman residual error, the solution found by TD has a better policy and is more robust against the perturbation of neural network parameters. We further use experiments to reveal a key difference between reinforcement learning and supervised learning, that is, a small Bellman residual error can correspond to a bad policy in reinforcement learning while the test loss function in supervised learning is a standard index to indicate the performance. We also empirically examine that the missing

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyin2022an

Related papers

• Neural Temporal-difference And Q-learning Provably Converge To Global Optima (2019)7.81
• Approximate Temporal Difference Learning Is A Gradient Descent For Reversible Policies (2018)0.00
• Gradient Temporal-difference Learning With Regularized Corrections (2020)0.00
• Simplifying Deep Temporal Difference Learning (2024)0.00
• On The Performance Of Temporal Difference Learning With Neural Networks (2023)0.00
• Nonlinear Distributional Gradient Temporal-difference Learning (2018)0.00
• An Improved Finite-time Analysis Of Temporal Difference Learning With Deep Neural Networks (2024)0.00
• Target-based Temporal Difference Learning (2019)0.00