Batch Reinforcement Learning From Crowds

Abstract

A shortcoming of batch reinforcement learning is its requirement for rewards in data, thus not applicable to tasks without reward functions. Existing settings for lack of reward, such as behavioral cloning, rely on optimal demonstrations collected from humans. Unfortunately, extensive expertise is required for ensuring optimality, which hinder the acquisition of large-scale data for complex tasks. This paper addresses the lack of reward in a batch reinforcement learning setting by learning a reward function from preferences. Generating preferences only requires a basic understanding of a task. Being a mental process, generating preferences is faster than performing demonstrations. So preferences can be collected at scale from non-expert humans using crowdsourcing. This paper tackles a critical challenge that emerged when collecting data from non-expert humans: the noise in preferences. A novel probabilistic model is proposed for modelling the reliability of labels, which utilizes label

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2021batch

Related papers

• Reinforcement Learning From Diverse Human Preferences (2023)0.00
• Hindsight Priors For Reward Learning From Human Preferences (2024)0.00
• Knowledge Transfer From Teachers To Learners In Growing-batch Reinforcement Learning (2023)0.00
• Data Driven Reward Initialization For Preference Based Reinforcement Learning (2023)0.00
• Adaptive Symmetric Reward Noising For Reinforcement Learning (2019)0.00
• Deep Reinforcement Learning From Hierarchical Preference Design (2023)2.00
• Causal Confusion And Reward Misidentification In Preference-based Reward Learning (2022)0.00
• Which Rewards Matter? Reward Selection For Reinforcement Learning Under Limited Feedback (2025)0.00