Episodic Return Decomposition By Difference Of Implicitly Assigned Sub-trajectory Reward

Abstract

Real-world decision-making problems are usually accompanied by delayed rewards, which affects the sample efficiency of Reinforcement Learning, especially in the extremely delayed case where the only feedback is the episodic reward obtained at the end of an episode. Episodic return decomposition is a promising way to deal with the episodic-reward setting. Several corresponding algorithms have shown remarkable effectiveness of the learned step-wise proxy rewards from return decomposition. However, these existing methods lack either attribution or representation capacity, leading to inefficient decomposition in the case of long-term episodes. In this paper, we propose a novel episodic return decomposition method called Diaster (Difference of implicitly assigned sub-trajectory reward). Diaster decomposes any episodic reward into credits of two divided sub-trajectories at any cut point, and the step-wise proxy rewards come from differences in expectation. We theoretically and empirically ve

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylin2023episodic

Related papers

• Learning Long-term Reward Redistribution Via Randomized Return Decomposition (2021)0.00
• RUDDER: Return Decomposition For Delayed Rewards (2018)0.00
• Rewarding Episodic Visitation Discrepancy For Exploration In Reinforcement Learning (2022)0.00
• Extending Differential Temporal Difference Methods For Episodic Problems (2026)0.00
• Improving The Efficiency Of Off-policy Reinforcement Learning By Accounting For Past Decisions (2021)0.00
• DIFFER: Decomposing Individual Reward For Fair Experience Replay In Multi-agent Reinforcement Learning (2023)2.26
• DEIR: Efficient And Robust Exploration Through Discriminative-model-based Episodic Intrinsic Rewards (2023)0.00
• Detecting Rewards Deterioration In Episodic Reinforcement Learning (2020)0.00