From Reinforcement Learning To Optimal Control: A Unified Framework For Sequential Decisions

Abstract

There are over 15 distinct communities that work in the general area of sequential decisions and information, often referred to as decisions under uncertainty or stochastic optimization. We focus on two of the most important fields: stochastic optimal control, with its roots in deterministic optimal control, and reinforcement learning, with its roots in Markov decision processes. Building on prior work, we describe a unified framework that covers all 15 different communities, and note the strong parallels with the modeling framework of stochastic optimal control. By contrast, we make the case that the modeling framework of reinforcement learning, inherited from discrete Markov decision processes, is quite limited. Our framework (and that of stochastic control) is based on the core problem of optimizing over policies. We describe four classes of policies that we claim are universal, and show that each of these two fields have, in their own way, evolved to include examples of each of the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypowell2019from

Related papers

• Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach (2023)0.00
• A General Markov Decision Process Framework For Directly Learning Optimal Control Policies (2019)0.00
• Reinforcement Learning And Control As Probabilistic Inference: Tutorial And Review (2018)0.00
• Unified Algorithms For RL With Decision-estimation Coefficients: PAC, Reward-free, Preference-based Learning, And Beyond (2022)5.24
• Reinforcement Learning In Economics And Finance (2020)14.73
• Learning Mdps From Features: Predict-then-optimize For Sequential Decision Problems By Reinforcement Learning (2021)0.00
• Optimal Decision-making In Mixed-agent Partially Observable Stochastic Environments Via Reinforcement Learning (2019)0.00
• A Tour Of Reinforcement Learning: The View From Continuous Control (2018)19.86