Non-markovian Reward Modelling From Trajectory Labels Via Interpretable Multiple Instance Learning

Abstract

We generalise the problem of reward modelling (RM) for reinforcement learning (RL) to handle non-Markovian rewards. Existing work assumes that human evaluators observe each step in a trajectory independently when providing feedback on agent behaviour. In this work, we remove this assumption, extending RM to capture temporal dependencies in human assessment of trajectories. We show how RM can be approached as a multiple instance learning (MIL) problem, where trajectories are treated as bags with return labels, and steps within the trajectories are instances with unseen reward labels. We go on to develop new MIL models that are able to capture the time dependencies in labelled trajectories. We demonstrate on a range of RL tasks that our novel MIL models can reconstruct reward functions to a high level of accuracy, and can be used to train high-performing agent policies.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyearly2022non

Related papers

• Aligning Agents Via Planning: A Benchmark For Trajectory-level Reward Modeling (2026)0.00
• Provable Multi-task Reinforcement Learning: A Representation Learning Framework With Low Rank Rewards (2026)0.00
• Learning Robust Reward Machines From Noisy Labels (2024)0.00
• Learning Multimodal Transition Dynamics For Model-based Reinforcement Learning (2017)0.00
• Reinforcement Learning From Human Feedback Without Reward Inference: Model-free Algorithm And Instance-dependent Analysis (2024)0.00
• Learning Reward Machines: A Study In Partially Observable Reinforcement Learning (2021)0.00
• Learning Symbolic Representations For Reinforcement Learning Of Non-markovian Behavior (2023)0.00
• Learning Task Automata For Reinforcement Learning Using Hidden Markov Models (2022)2.26