Revisiting State Augmentation Methods For Reinforcement Learning With Stochastic Delays

Abstract

Several real-world scenarios, such as remote control and sensing, are comprised of action and observation delays. The presence of delays degrades the performance of reinforcement learning (RL) algorithms, often to such an extent that algorithms fail to learn anything substantial. This paper formally describes the notion of Markov Decision Processes (MDPs) with stochastic delays and shows that delayed MDPs can be transformed into equivalent standard MDPs (without delays) with significantly simplified cost structure. We employ this equivalence to derive a model-free Delay-Resolved RL framework and show that even a simple RL algorithm built upon this framework achieves near-optimal rewards in environments with stochastic delays in actions and observations. The delay-resolved deep Q-network (DRDQN) algorithm is bench-marked on a variety of environments comprising of multi-step and stochastic delays and results in better performance, both in terms of achieving near-optimal rewards and minim

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keynath2021revisiting

Related papers

• Model-based Reinforcement Learning Under Random Observation Delays (2025)0.00
• Boosting Reinforcement Learning With Strongly Delayed Feedback Through Auxiliary Short Delays (2024)1.69
• Reinforcement Learning For Control Systems With Time Delays: A Comprehensive Survey (2026)0.00
• Learning Adversarial Markov Decision Processes With Delayed Feedback (2020)0.00
• Delay-aware Multi-agent Reinforcement Learning For Cooperative And Competitive Environments (2020)0.00
• Blind Decision Making: Reinforcement Learning With Delayed Observations (2020)0.00
• Reinforcement Learning With Random Delays (2020)0.00
• Reinforcement Learning With Delayed, Composite, And Partially Anonymous Reward (2023)0.00