Information Directed Reward Learning For Reinforcement Learning

Abstract

For many reinforcement learning (RL) applications, specifying a reward is difficult. This paper considers an RL setting where the agent obtains information about the reward only by querying an expert that can, for example, evaluate individual states or provide binary preferences over trajectories. From such expensive feedback, we aim to learn a model of the reward that allows standard RL algorithms to achieve high expected returns with as few expert queries as possible. To this end, we propose Information Directed Reward Learning (IDRL), which uses a Bayesian model of the reward and selects queries that maximize the information gain about the difference in return between plausibly optimal policies. In contrast to prior active reward learning methods designed for specific types of queries, IDRL naturally accommodates different query types. Moreover, it achieves similar or better performance with significantly fewer queries by shifting the focus from reducing the reward approximation err

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylindner2021information

Related papers

• Provably Feedback-efficient Reinforcement Learning Via Active Reward Learning (2023)0.00
• Inforl: Interpretable Reinforcement Learning Using Information Maximization (2019)0.00
• Distance-rank Aware Sequential Reward Learning For Inverse Reinforcement Learning With Sub-optimal Demonstrations (2023)0.00
• STEERING: Stein Information Directed Exploration For Model-based Reinforcement Learning (2023)0.00
• Task-guided Inverse Reinforcement Learning Under Partial Information (2021)0.00
• Inverse Reinforcement Learning Without Reinforcement Learning (2023)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• Internally Rewarded Reinforcement Learning (2023)0.00