A Hierarchical Bayesian Approach To Inverse Reinforcement Learning With Symbolic Reward Machines

Abstract

A misspecified reward can degrade sample efficiency and induce undesired behaviors in reinforcement learning (RL) problems. We propose symbolic reward machines for incorporating high-level task knowledge when specifying the reward signals. Symbolic reward machines augment existing reward machine formalism by allowing transitions to carry predicates and symbolic reward outputs. This formalism lends itself well to inverse reinforcement learning, whereby the key challenge is determining appropriate assignments to the symbolic values from a few expert demonstrations. We propose a hierarchical Bayesian approach for inferring the most likely assignments such that the concretized reward machine can discriminate expert demonstrated trajectories from other trajectories with high accuracy. Experimental results show that learned reward machines can significantly improve training efficiency for complex RL tasks and generalize well across different task environment configurations.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhou2022a

Related papers

• Learning Reward Machines: A Study In Partially Observable Reinforcement Learning (2021)0.00
• Learning Symbolic Representations For Reinforcement Learning Of Non-markovian Behavior (2023)0.00
• Contextual Pre-planning On Reward Machine Abstractions For Enhanced Transfer In Deep Reinforcement Learning (2023)2.26
• Joint Learning Of Reward Machines And Policies In Environments With Partially Known Semantics (2022)3.58
• Inferring Probabilistic Reward Machines From Non-markovian Reward Processes For Reinforcement Learning (2021)0.00
• Leveraging Approximate Symbolic Models For Reinforcement Learning Via Skill Diversity (2022)0.00
• Interpretable Model-based Hierarchical Reinforcement Learning Using Inductive Logic Programming (2021)0.00
• Learning Task Automata For Reinforcement Learning Using Hidden Markov Models (2022)2.26