TIC-GRPO: Provable And Efficient Optimization For Reinforcement Learning From Human Feedback

Abstract

Group Relative Policy Optimization (GRPO), recently introduced by DeepSeek, is a critic-free reinforcement learning algorithm for fine-tuning large language models. GRPO replaces the value function in Proximal Policy Optimization (PPO) with group-normalized rewards while retaining PPO-style token-level importance sampling based on an old policy. Our theoretical analysis reveals that the GRPO update rule estimates the policy gradient at the old policy rather than the current one; however, since the old policy is refreshed every few steps, the resulting discrepancy remains small and the induced bias is negligible in practice. To empirically validate this insight, we conduct an ablation study that entirely removes importance sampling and performs multiple optimization steps using gradients estimated at a fixed old policy. Remarkably, this simplified variant attains performance comparable to standard GRPO. Motivated by this finding, we propose Trajectory-level Importance-Corrected GRPO (

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypang2025tic

Related papers

• Hybrid Group Relative Policy Optimization: A Multi-sample Approach To Enhancing Policy Optimization (2025)0.00
• EP-GRPO: Entropy-progress Aligned Group Relative Policy Optimization With Implicit Process Guidance (2026)0.00
• Stepwise Guided Policy Optimization: Coloring Your Incorrect Reasoning In GRPO (2025)0.00
• Demystifying Group Relative Policy Optimization: Its Policy Gradient Is A U-statistic (2026)0.00
• PTR-PPO: Proximal Policy Optimization With Prioritized Trajectory Replay (2021)0.00
• Proximal Policy Optimization Algorithms (2017)0.00
• Noise-corrected GRPO: From Noisy Rewards To Unbiased Gradients (2025)0.00
• NGRPO: Negative-enhanced Group Relative Policy Optimization (2025)0.00