Unifying Value Iteration, Advantage Learning, And Dynamic Policy Programming

Abstract

Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper we propose a new, robust dynamic programming algorithm that unifies value iteration, advantage learning, and dynamic policy programming. We call it generalized value iteration (GVI) and its approximated version, approximate GVI (AGVI). We show AGVI's performance guarantee, which includes performance guarantees for existing algorithms, as special cases. We discuss theoretical weaknesses of existing algorithms, and explain the advantages of AGVI. Numerical experiments in a simple environment support theoretical arguments, and suggest that AGVI is a promising alternative to previous algorithms.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykozuno2017unifying

Related papers

• Adaptive Approximate Policy Iteration (2020)0.00
• The Role Of Lookahead And Approximate Policy Evaluation In Reinforcement Learning With Linear Value Function Approximation (2021)0.00
• Biased Aggregation, Rollout, And Enhanced Policy Improvement For Reinforcement Learning (2019)0.00
• Revisiting Value Iteration: Unified Analysis Of Discounted And Average-reward Cases (2025)0.00
• What About Inputing Policy In Value Function: Policy Representation And Policy-extended Value Function Approximator (2020)2.26
• Adaptive Exploration For Data-efficient General Value Function Evaluations (2024)0.00
• Linear Function Approximation As A Computationally Efficient Method To Solve Classical Reinforcement Learning Challenges (2024)0.00
• Deflated Dynamics Value Iteration (2024)0.00