Direct And Indirect Reinforcement Learning

Abstract

Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal policy by directly maximizing an objective function using gradient descent methods, in which the objective function is usually the expectation of accumulative future rewards. The latter indirectly finds the optimal policy by solving the Bellman equation, which is the sufficient and necessary condition from Bellman's principle of optimality. We study policy gradient forms of direct and indirect RL and show that both of them can derive the actor-critic architecture and can be unified into a policy gradient with the approximate value function and the stationary state distribution, revealing the equivalence of direct and indirect RL. We employ a Gridworld task to verify the influ

Stats

• citations26
• S2 citations—
• github stars0
• HF likes0
• heat score10.74
• arxiv keyguan2019direct

Related papers

• A General Markov Decision Process Framework For Directly Learning Optimal Control Policies (2019)0.00
• Natural Policy Gradient And Actor Critic Methods For Constrained Multi-task Reinforcement Learning (2024)0.00
• Policy Gradient RL Algorithms As Directed Acyclic Graphs (2020)0.00
• Automated Reinforcement Learning: An Overview (2022)0.00
• Discovering Reinforcement Learning Algorithms (2020)0.00
• Policy Gradient For Reinforcement Learning With General Utilities (2022)0.00
• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• Programmatic Reinforcement Learning: Navigating Gridworlds (2024)0.00