How To Discretize Continuous State-action Spaces In Q-learning: A Symbolic Control Approach

Abstract

Q-learning is widely recognized as an effective approach for synthesizing controllers to achieve specific goals. However, handling challenges posed by continuous state-action spaces remains an ongoing research focus. This paper presents a systematic analysis that highlights a major drawback in space discretization methods. To address this challenge, the paper proposes a symbolic model that represents behavioral relations, such as alternating simulation from abstraction to the controlled system. This relation allows for seamless application of the synthesized controller based on abstraction to the original system. Introducing a novel Q-learning technique for symbolic models, the algorithm yields two Q-tables encoding optimal policies. Theoretical analysis demonstrates that these Q-tables serve as both upper and lower bounds on the Q-values of the original system with continuous spaces. Additionally, the paper explores the correlation between the parameters of the space abstraction and t

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyalaoui2024how

Related papers

• Learning State Abstractions For Transfer In Continuous Control (2020)0.00
• Solving Continuous Control Via Q-learning (2022)0.00
• Symbolic State Partitioning For Reinforcement Learning (2024)0.00
• Adaptive Discretization For Episodic Reinforcement Learning In Metric Spaces (2019)2.26
• Specialized Deep Residual Policy Safe Reinforcement Learning-based Controller For Complex And Continuous State-action Spaces (2023)4.52
• Q-learning In Continuous Time (2022)0.00
• Continuous-time Q-learning For Mean-field Control Problems (2023)0.00
• On The Geometry Of Reinforcement Learning In Continuous State And Action Spaces (2022)0.00