Reinforcement Learning From Human Feedback Without Reward Inference: Model-free Algorithm And Instance-dependent Analysis

Abstract

In this paper, we study reinforcement learning from human feedback (RLHF) under an episodic Markov decision process with a general trajectory-wise reward model. We developed a model-free RLHF best policy identification algorithm, called \(\mathsf\{BSAD\}\), without explicit reward model inference, which is a critical intermediate step in the contemporary RLHF paradigms for training large language models (LLM). The algorithm identifies the optimal policy directly from human preference information in a backward manner, employing a dueling bandit sub-routine that constantly duels actions to identify the superior one. \(\mathsf\{BSAD\}\) adopts a reward-free exploration and best-arm-identification-like adaptive stopping criteria to equalize the visitation among all states in the same decision step while moving to the previous step as soon as the optimal action is identifiable, leading to a provable, instance-dependent sample complexity \(\tilde\{\mathcal\{O\}\}(c_\{\mathcal\{M\}\}SA^3H^3Ml

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2024reinforcement

Related papers

• A Survey Of Reinforcement Learning From Human Feedback (2023)0.00
• Principled Reinforcement Learning With Human Feedback From Pairwise Or \(k\)-wise Comparisons (2023)0.00
• Provably Feedback-efficient Reinforcement Learning Via Active Reward Learning (2023)0.00
• Zeroth-order Policy Gradient For Reinforcement Learning From Human Feedback Without Reward Inference (2024)0.00
• Reinforcement Learning With Human Feedback: Learning Dynamic Choices Via Pessimism (2023)0.00
• Exploration-driven Policy Optimization In RLHF: Theoretical Insights On Efficient Data Utilization (2024)0.00
• Reward-free Model-based Reinforcement Learning With Linear Function Approximation (2021)0.00
• Making RL With Preference-based Feedback Efficient Via Randomization (2023)0.00