Reinforcement Learning For Control Systems With Time Delays: A Comprehensive Survey

Abstract

In the last decade, Reinforcement Learning (RL) has achieved remarkable success in the control and decision-making of complex dynamical systems. However, most RL algorithms rely on the Markov Decision Process assumption, which is violated in practical cyber-physical systems affected by sensing delays, actuation latencies, and communication constraints. Such time delays introduce memory effects that can significantly degrade performance and compromise stability, particularly in networked and multi-agent environments. This paper presents a comprehensive survey of RL methods designed to address time delays in control systems. We first formalize the main classes of delays and analyze their impact on the Markov property. We then systematically categorize existing approaches into five major families: state augmentation and history-based representations, recurrent policies with learned memory, predictor-based and model-aware methods, robust and domain-randomized training strategies, and safe

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyneto2026reinforcement

Related papers

• Revisiting State Augmentation Methods For Reinforcement Learning With Stochastic Delays (2021)10.35
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• When To Sense And Control? A Time-adaptive Approach For Continuous-time RL (2024)0.00
• Model-based Reinforcement Learning Under Random Observation Delays (2025)0.00
• Deep Reinforcement Learning Based Networked Control With Network Delays For Signal Temporal Logic Specifications (2021)0.00
• Delay-aware Multi-agent Reinforcement Learning For Cooperative And Competitive Environments (2020)0.00
• Reinforcement Learning Via Conservative Agent For Environments With Random Delays (2025)0.00
• A Comprehensive Survey Of Reinforcement Learning: From Algorithms To Practical Challenges (2024)0.00