Learning Long-term Reward Redistribution Via Randomized Return Decomposition

Abstract

Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider the problem formulation of episodic reinforcement learning with trajectory feedback. It refers to an extreme delay of reward signals, in which the agent can only obtain one reward signal at the end of each trajectory. A popular paradigm for this problem setting is learning with a designed auxiliary dense reward function, namely proxy reward, instead of sparse environmental signals. Based on this framework, this paper proposes a novel reward redistribution algorithm, randomized return decomposition (RRD), to learn a proxy reward function for episodic reinforcement learning. We establish a surrogate problem by Monte-Carlo sampling that scales up least-squares-based reward redistribution to long-horizon problems. We analyze our surrogate loss function by connection with e

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyren2021learning

Related papers

• Episodic Return Decomposition By Difference Of Implicitly Assigned Sub-trajectory Reward (2023)0.00
• Interpretable Reward Redistribution In Reinforcement Learning: A Causal Approach (2023)2.26
• RUDDER: Return Decomposition For Delayed Rewards (2018)0.00
• Learning Self-imitating Diverse Policies (2018)0.00
• Monte Carlo Augmented Actor-critic For Sparse Reward Deep Reinforcement Learning From Suboptimal Demonstrations (2022)0.00
• Provably Efficient Offline Reinforcement Learning With Trajectory-wise Reward (2022)0.00
• Reward Shaping With Dynamic Trajectory Aggregation (2021)0.00
• Inverse Reinforcement Learning From Non-stationary Learning Agents (2024)0.00