A Kernel Loss For Solving The Bellman Equation

Abstract

Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variant of Bellman operator that is not necessarily a contraction. As a result, they may easily lose convergence guarantees, as can be observed in practice. In this paper, we propose a novel loss function, which can be optimized using standard gradient-based methods without risking divergence. The key advantage is that its gradient can be easily approximated using sampled transitions, avoiding the need for double samples required by prior algorithms like residual gradient. Our approach may be combined with general function classes such as neural networks, on either on- or off-policy data, and is shown to work reliably and effectively in several benchmarks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfeng2019a

Related papers

• Robust Losses For Learning Value Functions (2022)0.00
• SBEED: Convergent Reinforcement Learning With Nonlinear Function Approximation (2017)0.00
• Symmetric Q-learning: Reducing Skewness Of Bellman Error In Online Reinforcement Learning (2024)0.00
• Reinforcement Learning With Non-cumulative Objective (2023)5.24
• Is The Bellman Residual A Bad Proxy? (2016)0.00
• A Generalized Projected Bellman Error For Off-policy Value Estimation In Reinforcement Learning (2021)0.00
• Continuous-time Reinforcement Learning: Ellipticity Enables Model-free Value Function Approximation (2026)0.00
• Learning Value Functions In Deep Policy Gradients Using Residual Variance (2020)0.00