Scores As Actions: A Framework Of Fine-tuning Diffusion Models By Continuous-time Reinforcement Learning

Abstract

Reinforcement Learning from human feedback (RLHF) has been shown a promising direction for aligning generative models with human intent and has also been explored in recent works for alignment of diffusion generative models. In this work, we provide a rigorous treatment by formulating the task of fine-tuning diffusion models, with reward functions learned from human feedback, as an exploratory continuous-time stochastic control problem. Our key idea lies in treating the score-matching functions as controls/actions, and upon this, we develop a unified framework from a continuous-time perspective, to employ reinforcement learning (RL) algorithms in terms of improving the generation quality of diffusion models. We also develop the corresponding continuous-time RL theory for policy optimization and regularization under assumptions of stochastic different equations driven environment. Experiments on the text-to-image (T2I) generation will be reported in the accompanied paper.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2024scores

Related papers

• Reward-directed Score-based Diffusion Models Via Q-learning (2024)0.00
• Learning A Diffusion Model Policy From Rewards Via Q-score Matching (2023)0.00
• Contractive Diffusion Policies: Robust Action Diffusion Via Contractive Score-based Sampling With Differential Equations (2026)0.00
• Using Human Feedback To Fine-tune Diffusion Models Without Any Reward Model (2023)17.39
• Avoiding Mode Collapse In Diffusion Models Fine-tuned With Reinforcement Learning (2024)0.00
• Goal-driven Reward By Video Diffusion Models For Reinforcement Learning (2025)0.00
• Diffusion Models For Reinforcement Learning: A Survey (2023)5.64
• Evolving Diffusion And Flow Matching Policies For Online Reinforcement Learning (2025)0.00