Iterated \(q\)-network: Beyond One-step Bellman Updates In Deep Reinforcement Learning

Abstract

The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated \(Q\)-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence o

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvincent2024iterated

Related papers

• Elastic Step DQN: A Novel Multi-step Algorithm To Alleviate Overestimation In Deep Qnetworks (2022)10.85
• Adaptive \(q\)-network: On-the-fly Target Selection For Deep Reinforcement Learning (2024)0.00
• Residual Q-networks For Value Function Factorizing In Multi-agent Reinforcement Learning (2022)10.21
• Multi-step Reinforcement Learning: A Unifying Algorithm (2017)12.68
• Nonparametric Bellman Mappings For Value Iteration In Distributed Reinforcement Learning (2025)0.00
• Enhancing Q-value Updates In Deep Q-learning Via Successor-state Prediction (2025)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Understanding Multi-step Deep Reinforcement Learning: A Systematic Study Of The DQN Target (2019)0.00