Overcoming Slow Decision Frequencies In Continuous Control: Model-based Sequence Reinforcement Learning For Model-free Control

Abstract

Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is impractical in real-world settings and typically necessitates specialized hardware. We introduce Sequence Reinforcement Learning (SRL), an RL algorithm designed to produce a sequence of actions for a given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both a model and an actor-critic architecture operating at different temporal scales. We propose a "temporal recall" mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free contro

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypatel2024overcoming

Related papers

• When To Sense And Control? A Time-adaptive Approach For Continuous-time RL (2024)0.00
• Temporal Difference Models: Model-free Deep RL For Model-based Control (2018)0.00
• Reinforcement Learning For Control Systems With Time Delays: A Comprehensive Survey (2026)0.00
• Deep RL With Information Constrained Policies: Generalization In Continuous Control (2020)0.00
• ACERAC: Efficient Reinforcement Learning In Fine Time Discretization (2021)4.52
• Model-free Reinforcement Learning For Model-based Control: Towards Safe, Interpretable And Sample-efficient Agents (2025)0.00
• Control-optimized Deep Reinforcement Learning For Artificially Intelligent Autonomous Systems (2025)0.00
• Automated Reinforcement Learning: An Overview (2022)0.00