A Comparative Analysis Of Expected And Distributional Reinforcement Learning

Abstract

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the reasons behind the improvements distributional RL provides. In this paper we begin the investigation into this fundamental question by analyzing the differences in the tabular, linear approximation, and non-linear approximation settings. We prove that in many realizations of the tabular and linear approximation settings, distributional RL behaves exactly the same as expected RL. In cases where the two methods behave differently, distributional RL can in fact hurt performance when it does not induce identical behaviour. We then continue with an empirical analysis comparing distributional and expected RL methods in control settings with non-linear approximators to tease apart

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keylyle2019a

Related papers

• A Differential Perspective On Distributional Reinforcement Learning (2025)0.00
• A Distributional Perspective On Reinforcement Learning (2017)0.00
• Assessing The Impact Of Distribution Shift On Reinforcement Learning Performance (2024)0.00
• Exploring The Training Robustness Of Distributional Reinforcement Learning Against Noisy State Observations (2021)0.00
• Normality-guided Distributional Reinforcement Learning For Continuous Control (2022)0.00
• Distributional Reinforcement Learning With Dual Expectile-quantile Regression (2023)0.00
• Action Gaps And Advantages In Continuous-time Distributional Reinforcement Learning (2024)0.00
• Distributional Reinforcement Learning With Linear Function Approximation (2019)0.00