Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces

Abstract

We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces, and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two key stability properties, relating to how changes in value functions and/or policies affect the Bellman operator and occupation measures. We argue that these properties are satisfied in many continuous state-action Markov decision processes, and demonstrate how they arise naturally when using linear function approximation methods. Our analysis offers fresh perspectives on the roles of pessimism and optimism in off-line and on-line RL, and highlights the connection between off-line RL and transfer learning.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyduan2024taming

Related papers

• Learning To Stabilize Online Reinforcement Learning In Unbounded State Spaces (2023)0.00
• Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearity (2021)0.00
• Policy Optimization For Continuous Reinforcement Learning (2023)2.26
• Stable Reinforcement Learning With Unbounded State Space (2020)0.00
• Policy Optimization Over General State And Action Spaces (2022)0.00
• On The Geometry Of Reinforcement Learning In Continuous State And Action Spaces (2022)0.00
• SALSA-RL: Stability Analysis In The Latent Space Of Actions For Reinforcement Learning (2025)0.00
• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00