Bellman Diffusion: Generative Modeling As Learning A Linear Operator In The Distribution Space

Abstract

Deep Generative Models (DGMs), including Energy-Based Models (EBMs) and Score-based Generative Models (SGMs), have advanced high-fidelity data generation and complex continuous distribution approximation. However, their application in Markov Decision Processes (MDPs), particularly in distributional Reinforcement Learning (RL), remains underexplored, with conventional histogram-based methods dominating the field. This paper rigorously highlights that this application gap is caused by the nonlinearity of modern DGMs, which conflicts with the linearity required by the Bellman equation in MDPs. For instance, EBMs involve nonlinear operations such as exponentiating energy functions and normalizing constants. To address this, we introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. With divergence-based training techniques to optimize neural network proxies and a new type of stochastic differential equation (SDE) for s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2024bellman

Related papers

• Maximum Entropy Inverse Reinforcement Learning Of Diffusion Models With Energy-based Models (2024)0.00
• Reward-directed Score-based Diffusion Models Via Q-learning (2024)0.00
• Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective (2024)0.00
• Distributional Hamilton-jacobi-bellman Equations For Continuous-time Reinforcement Learning (2022)0.00
• MDPO: Overcoming The Training-inference Divide Of Masked Diffusion Language Models (2025)0.00
• Sampling From Energy-based Policies Using Diffusion (2024)0.00
• The Curious Price Of Distributional Robustness In Reinforcement Learning With A Generative Model (2023)0.00
• Genpo: Generative Diffusion Models Meet On-policy Reinforcement Learning (2025)0.00