Diffpo: Training Diffusion Llms To Reason Fast And Furious Via Reinforcement Learning

Abstract

We propose DiFFPO, Diffusion Fast and Furious Policy Optimization, a unified framework for training masked diffusion large language models (dLLMs) to reason not only better (furious), but also faster via reinforcement learning (RL). We first unify the existing baseline approach such as d1 by proposing to train surrogate policies via off-policy RL, whose likelihood is much more tractable as an approximation to the true dLLM policy. This naturally motivates a more accurate and informative two-stage likelihood approximation combined with importance sampling correction, which leads to generalized RL algorithms with better sample efficiency and superior task performance. Second, we propose a new direction of joint training efficient samplers/controllers of dLLMs policy. Via RL, we incentivize dLLMs' natural multi-token prediction capabilities by letting the model learn to adaptively allocate an inference threshold for each prompt. By jointly training the sampler, we yield better accuracies

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2025diffpo

Related papers

• Wd1: Weighted Policy Optimization For Reasoning In Diffusion Language Models (2025)0.00
• Simple Policy Gradients For Reasoning With Diffusion Language Models (2025)0.00
• Diffusion Policy Through Conditional Proximal Policy Optimization (2026)0.00
• Dichotomous Diffusion Policy Optimization (2025)0.00
• Diffusionnft: Online Diffusion Reinforcement With Forward Process (2025)0.00
• Policy Representation Via Diffusion Probability Model For Reinforcement Learning (2023)0.00
• Diffusion Policies As An Expressive Policy Class For Offline Reinforcement Learning (2022)0.00
• Inpainting-guided Policy Optimization For Diffusion Large Language Models (2025)0.00