AEGPO: Adaptive Entropy-guided Policy Optimization For Diffusion Models

Abstract

Reinforcement learning from human feedback (RLHF) shows promise for aligning diffusion and flow models, yet policy optimization methods such as GRPO suffer from inefficient and static sampling strategies. These methods treat all prompts and denoising steps uniformly, ignoring substantial variations in sample learning value as well as the dynamic nature of critical exploration moments. To address this issue, we conduct a detailed analysis of the internal attention dynamics during GRPO training and uncover a key insight: attention entropy can serve as a powerful dual-signal proxy. First, across different samples, the relative change in attention entropy (\{\Delta\}Entropy), which reflects the divergence between the current policy and the base policy, acts as a robust indicator of sample learning value. Second, during the denoising process, the peaks of absolute attention entropy (Entropy(t)), which quantify attention dispersion, effectively identify critical timesteps where high-value

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2026aegpo

Related papers

• EP-GRPO: Entropy-progress Aligned Group Relative Policy Optimization With Implicit Process Guidance (2026)0.00
• Arbitrary Entropy Policy Optimization Breaks The Exploration Bottleneck Of Reinforcement Learning (2025)0.00
• Genpo: Generative Diffusion Models Meet On-policy Reinforcement Learning (2025)0.00
• Dichotomous Diffusion Policy Optimization (2025)0.00
• Fine-tuning Diffusion Policies With Backpropagation Through Diffusion Timesteps (2025)0.00
• TIC-GRPO: Provable And Efficient Optimization For Reinforcement Learning From Human Feedback (2025)0.00
• Stabilizing Reinforcement Learning For Diffusion Language Models (2026)0.00
• Evolving Diffusion And Flow Matching Policies For Online Reinforcement Learning (2025)0.00