Time Limits In Reinforcement Learning

Abstract

In reinforcement learning, it is common to let an agent interact for a fixed amount of time with its environment before resetting it and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed period, or (ii) an indefinite period where time limits are only used during training to diversify experience. In this paper, we provide a formal account for how time limits could effectively be handled in each of the two cases and explain why not doing so can cause state aliasing and invalidation of experience replay, leading to suboptimal policies and training instability. In case (i), we argue that the terminations due to time limits are in fact part of the environment, and thus a notion of the remaining time should be included as part of the agent's input to avoid violation of the Markov property. In case (ii), the time limits are not part of the environment and are only used to facilitate learning. We ar

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypardo2017time

Related papers

• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• Interval Timing In Deep Reinforcement Learning Agents (2019)0.00
• Tempo Adaptation In Non-stationary Reinforcement Learning (2023)0.00
• Optimal Options For Multi-task Reinforcement Learning Under Time Constraints (2020)0.00
• An Idiosyncrasy Of Time-discretization In Reinforcement Learning (2024)0.00
• Continual Learning In Environments With Polynomial Mixing Times (2021)0.00
• Deterministic Limit Of Temporal Difference Reinforcement Learning For Stochastic Games (2018)12.93
• Ecological Reinforcement Learning (2020)8.35