Distributional Reinforcement Learning With Regularized Wasserstein Loss

Abstract

The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose \textit\{Sinkhorn distributional RL (SinkhornDRL)\}, which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysun2022distributional

Related papers

• A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning (2024)0.00
• Distributional Robustness And Regularization In Reinforcement Learning (2020)0.00
• Intrinsic Benefits Of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration In Reinforcement Learning (2021)0.00
• Distributional Reinforcement Learning For Multi-dimensional Reward Functions (2021)0.00
• A Differential Perspective On Distributional Reinforcement Learning (2025)0.00
• A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76
• Exploring The Training Robustness Of Distributional Reinforcement Learning Against Noisy State Observations (2021)0.00
• Federated Distributional Reinforcement Learning With Distributional Critic Regularization (2026)0.00