Wd1: Weighted Policy Optimization For Reasoning In Diffusion Language Models

Abstract

Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization step. This reliance introduces additional computational overhead, and can lead to large variance and estimation error in RL objective -- particularly in computing the policy ratio for importance sampling. To mitigate these issues, we introduce wd1, a novel ratio-free policy optimization approach that reformulates the RL objective as a weighted log-likelihood, requiring only a single approximation for the current parametrized policy likelihood. We formally show that our proposed method can be interpreted as energy-guided discrete diffusion training combined with negative sample unlearning, thereby confirming its theoretical soundness. In experiments on LLaDA-8B model, wd1 outperforms diffusi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2025wd1

Related papers

• Simple Policy Gradients For Reasoning With Diffusion Language Models (2025)0.00
• Diffpo: Training Diffusion Llms To Reason Fast And Furious Via Reinforcement Learning (2025)0.00
• Diffusion Policy Through Conditional Proximal Policy Optimization (2026)0.00
• Dichotomous Diffusion Policy Optimization (2025)0.00
• Stabilizing Reinforcement Learning For Diffusion Language Models (2026)0.00
• Policy Representation Via Diffusion Probability Model For Reinforcement Learning (2023)0.00
• Diwa: Diffusion Policy Adaptation With World Models (2025)0.00
• SPG: Sandwiched Policy Gradient For Masked Diffusion Language Models (2025)0.00