Accounting For Human Learning When Inferring Human Preferences

Abstract

Inverse reinforcement learning (IRL) is a common technique for inferring human preferences from data. Standard IRL techniques tend to assume that the human demonstrator is stationary, that is that their policy \(\pi\) doesn't change over time. In practice, humans interacting with a novel environment or performing well on a novel task will change their demonstrations as they learn more about the environment or task. We investigate the consequences of relaxing this assumption of stationarity, in particular by modelling the human as learning. Surprisingly, we find in some small examples that this can lead to better inference than if the human was stationary. That is, by observing a demonstrator who is themselves learning, a machine can infer more than by observing a demonstrator who is noisily rational. In addition, we find evidence that misspecification can lead to poor inference, suggesting that modelling human learning is important, especially when the human is facing an unfamiliar env

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygiles2020accounting

Related papers

• Misspecification In Inverse Reinforcement Learning (2022)5.24
• Reward Shaping For Human Learning Via Inverse Reinforcement Learning (2020)0.00
• Modeling And Interpreting Real-world Human Risk Decision Making With Inverse Reinforcement Learning (2019)0.00
• Partial Identifiability And Misspecification In Inverse Reinforcement Learning (2024)0.00
• Active Learning For Risk-sensitive Inverse Reinforcement Learning (2019)0.00
• Implications Of Human Irrationality For Reinforcement Learning (2020)0.00
• Can You See How I Learn? Human Observers' Inferences About Reinforcement Learning Agents' Learning Processes (2025)0.00
• Basis For Intentions: Efficient Inverse Reinforcement Learning Using Past Experience (2022)0.00