Distributional Hamilton-jacobi-bellman Equations For Continuous-time Reinforcement Learning

Abstract

Continuous-time reinforcement learning offers an appealing formalism for describing control problems in which the passage of time is not naturally divided into discrete increments. Here we consider the problem of predicting the distribution of returns obtained by an agent interacting in a continuous-time, stochastic environment. Accurate return predictions have proven useful for determining optimal policies for risk-sensitive control, learning state representations, multiagent coordination, and more. We begin by establishing the distributional analogue of the Hamilton-Jacobi-Bellman (HJB) equation for It\^o diffusions and the broader class of Feller-Dynkin processes. We then specialize this equation to the setting in which the return distribution is approximated by \(N\) uniformly-weighted particles, a common design choice in distributional algorithms. Our derivation highlights additional terms due to statistical diffusivity which arise from the proper handling of distributions in the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywiltzer2022distributional

Related papers

• Tractable Representations For Convergent Approximation Of Distributional HJB Equations (2025)0.00
• Conjugated Discrete Distributions For Distributional Reinforcement Learning (2021)0.00
• A Distributional Perspective On Reinforcement Learning (2017)0.00
• Continuous-time Reinforcement Learning: Ellipticity Enables Model-free Value Function Approximation (2026)0.00
• A Distributional Analysis Of Sampling-based Reinforcement Learning Algorithms (2020)0.00
• Distributional Soft Actor-critic With Diffusion Policy (2025)0.00
• Bellman Diffusion: Generative Modeling As Learning A Linear Operator In The Distribution Space (2024)0.00
• Action Gaps And Advantages In Continuous-time Distributional Reinforcement Learning (2024)0.00