\(V_{0.5}\): Generalist Value Model As A Prior For Sparse RL Rollouts

Abstract

In Reinforcement Learning with Verifiable Rewards (RLVR), constructing a robust advantage baseline is critical for policy gradients, effectively guiding the policy model to reinforce desired behaviors. Recent research has introduced Generalist Value Models (such as \(V_0\)), which achieve pre-trained value estimation by explicitly encoding model capabilities in-context, eliminating the need to synchronously update the value model alongside the policy model. In this paper, we propose \(V_\{0.5\}\), which adaptively fuses the baseline predicted by such value model (acting as a prior) with the empirical mean derived from sparse rollouts. This constructs a robust baseline that balances computational efficiency with extremely low variance. Specifically, we introduce a real-time statistical testing and dynamic budget allocation. This balances the high variance caused by sparse sampling against the systematic bias (or hallucinations) inherent in the value model's prior. By constructing a hypo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2026v_

Related papers

• Shrinking The Variance: Shrinkage Baselines For Reinforcement Learning With Verifiable Rewards (2025)0.00
• Pretrain Value, Not Reward: Decoupled Value Policy Optimization (2025)0.00
• Viva: Video-trained Value Functions For Guiding Online RL From Diverse Data (2025)0.00
• Tensor And Matrix Low-rank Value-function Approximation In Reinforcement Learning (2022)7.81
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• Rethinking Value Function Learning For Generalization In Reinforcement Learning (2022)0.00
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77