Verifiable And Compositional Reinforcement Learning Systems

Abstract

We propose a framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL subsystems, each of which learns to accomplish a separate subtask, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of subsystems, and of the collection of low-level subsystems themselves. By defining interfaces between the subsystems, the framework enables automatic decompositions of task specifications, e.g., reach a target set of states with a probability of at least 0.95, into individual subtask specifications, i.e. achieve the subsystem's exit conditions with at least some minimum probability, given that its entry conditions are met. This in turn allows for the independent training and testing of the subsystems; if they each learn a policy satisfying the appropriate subtask specification, then their composition is guaranteed t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyneary2021verifiable

Related papers

• Sample Efficient Reinforcement Learning By Automatically Learning To Compose Subtasks (2024)0.00
• Hierarchical Programmatic Reinforcement Learning Via Learning To Compose Programs (2023)0.00
• Unifying Task Specification In Reinforcement Learning (2016)0.00
• An Abstraction-based Method To Check Multi-agent Deep Reinforcement-learning Behaviors (2021)2.26
• Self-organization Of Action Hierarchy And Compositionality By Reinforcement Learning With Recurrent Neural Networks (2019)8.60
• Utilizing Prior Solutions For Reward Shaping And Composition In Entropy-regularized Reinforcement Learning (2022)3.58
• Provable Multi-task Reinforcement Learning: A Representation Learning Framework With Low Rank Rewards (2026)0.00
• Probabilistic Model Checking Of Stochastic Reinforcement Learning Policies (2024)0.00