A Regret Minimization Framework on Preference Learning in Large Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization $(\text{RePO})$, which reframes RLHF through $\text{regret minimization}$ rather than reward maximization. Human preferences are often shaped by $\text{prospective}$ anticipation of outcomes and $\text{counterfactual}$ comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. $\text{RePO}$ captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Experiments on mathematical reasoning benchmarks and human preference datasets demonstrate consistent performance gains, indicating that $\text{RePO}$ is an effective and human-aligned approach for training large language models.