Reinforcement Learning With Verifiable Rewards: Grpo's Effective Loss, Dynamics, And Success Amplification

Abstract

Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model \(\pi_\{\mathrm\{ref\}\}\), or combining both forms of regularization. For each, the optimal policy \(\pi_n\) admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under \(\pi_\{n-1\}\), as well as the policies \(\pi_\{n-1\}\) and \(\pi_\{\mathrm\{ref\}\}\). Iterating results in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymroueh2025reinforcement

Related papers

• Noise-corrected GRPO: From Noisy Rewards To Unbiased Gradients (2025)0.00
• EP-GRPO: Entropy-progress Aligned Group Relative Policy Optimization With Implicit Process Guidance (2026)0.00
• TIC-GRPO: Provable And Efficient Optimization For Reinforcement Learning From Human Feedback (2025)0.00
• A Unified Framework For Rethinking Policy Divergence Measures In GRPO (2026)0.00
• Group-relative REINFORCE Is Secretly An Off-policy Algorithm: Demystifying Some Myths About GRPO And Its Friends (2025)0.00
• NGRPO: Negative-enhanced Group Relative Policy Optimization (2025)0.00
• Stepwise Guided Policy Optimization: Coloring Your Incorrect Reasoning In GRPO (2025)0.00
• Think Outside The Policy: In-context Steered Policy Optimization (2025)0.00