Distributional Reinforcement Learning With Dual Expectile-quantile Regression

Abstract

Distributional reinforcement learning (RL) has proven useful in multiple benchmarks as it enables approximating the full distribution of returns and extracts rich feedback from environment samples. The commonly used quantile regression approach to distributional RL -- based on asymmetric \(L_1\) losses -- provides a flexible and effective way of learning arbitrary return distributions. In practice, it is often improved by using a more efficient, asymmetric hybrid \(L_1\)-\(L_2\) Huber loss for quantile regression. However, by doing so, distributional estimation guarantees vanish, and we empirically observe that the estimated distribution rapidly collapses to its mean. Indeed, asymmetric \(L_2\) losses, corresponding to expectile regression, cannot be readily used for distributional temporal difference. Motivated by the efficiency of \(L_2\)-based learning, we propose to jointly learn expectiles and quantiles of the return distribution in a way that allows efficient learning while keepi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjullien2023distributional

Related papers

• Distributional Reinforcement Learning With Quantile Regression (2017)19.20
• A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning (2024)0.00
• Fully Parameterized Quantile Function For Distributional Reinforcement Learning (2019)0.00
• A Differential Perspective On Distributional Reinforcement Learning (2025)0.00
• A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76
• Distributional Reinforcement Learning For Multi-dimensional Reward Functions (2021)0.00
• The Statistical Benefits Of Quantile Temporal-difference Learning For Value Estimation (2023)0.00
• Non-decreasing Quantile Function Network With Efficient Exploration For Distributional Reinforcement Learning (2021)4.52