Policy Representation Via Diffusion Probability Model For Reinforcement Learning

Abstract

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration. The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL. In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL. Concretely, we character diffusion policy as a stochastic process, which is a new approach to representing a policy. Then we present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy. Furthermore, we propose the DIPO which is an implementation for model-free online RL with DIffusion POlicy. To the best of our knowledge, DIPO is the first algorithm to solve model-free online RL probl

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2023policy

Related papers

• Diffusion Policies As An Expressive Policy Class For Offline Reinforcement Learning (2022)0.00
• Diffusion Policy Through Conditional Proximal Policy Optimization (2026)0.00
• Dichotomous Diffusion Policy Optimization (2025)0.00
• Preferred-action-optimized Diffusion Policies For Offline Reinforcement Learning (2024)0.00
• Genpo: Generative Diffusion Models Meet On-policy Reinforcement Learning (2025)0.00
• Diffusion Policies Creating A Trust Region For Offline Reinforcement Learning (2024)8.04
• How Does The Lagrangian Guide Safe Reinforcement Learning Through Diffusion Models? (2026)0.00
• Reverse Flow Matching: A Unified Framework For Online Reinforcement Learning With Diffusion And Flow Policies (2026)0.00