When To Sense And Control? A Time-adaptive Approach For Continuous-time RL

Abstract

Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs an inexact modeling choice. In many applications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, we generally prefer a time-adaptive approach with fewer interactions with the system. In this work, we formalize an RL framework, Time-adaptive Control & Sensing (TaCoS), that tackles this challenge by optimizing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve. We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhib

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytreven2024when

Related papers

• Reinforcement Learning For Control Systems With Time Delays: A Comprehensive Survey (2026)0.00
• Managing Temporal Resolution In Continuous Value Estimation: A Fundamental Trade-off (2022)0.00
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• ACERAC: Efficient Reinforcement Learning In Fine Time Discretization (2021)4.52
• Policy Optimization For Continuous Reinforcement Learning (2023)2.26
• Overcoming Slow Decision Frequencies In Continuous Control: Model-based Sequence Reinforcement Learning For Model-free Control (2024)0.00
• Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces (2024)2.26
• Deep RL With Information Constrained Policies: Generalization In Continuous Control (2020)0.00