Online Iterative Reinforcement Learning From Human Feedback With General Preference Model

Abstract

We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyye2024online

Related papers

• Principled Reinforcement Learning With Human Feedback From Pairwise Or \(k\)-wise Comparisons (2023)0.00
• A Survey Of Reinforcement Learning From Human Feedback (2023)0.00
• Making RL With Preference-based Feedback Efficient Via Randomization (2023)0.00
• Reinforcement Learning From Diverse Human Preferences (2023)0.00
• A General Theoretical Paradigm To Understand Learning From Human Preferences (2023)0.00
• Data-dependent Exploration For Online Reinforcement Learning From Human Feedback (2026)0.00
• Reinforcement Learning From Human Feedback Without Reward Inference: Model-free Algorithm And Instance-dependent Analysis (2024)0.00
• Reward Model Learning Vs. Direct Policy Optimization: A Comparative Analysis Of Learning From Human Preferences (2024)0.00