Stable Reinforcement Learning With Unbounded State Space

Abstract

We consider the problem of reinforcement learning (RL) with unbounded state space motivated by the classical problem of scheduling in a queueing network. Traditional policies as well as error metric that are designed for finite, bounded or compact state space, require infinite samples for providing any meaningful performance guarantee (e.g. \(\ell_\infty\) error) for unbounded state space. That is, we need a new notion of performance metric. As the main contribution of this work, inspired by the literature in queuing systems and control theory, we propose stability as the notion of "goodness": the state dynamics under the policy should remain in a bounded region with high probability. As a proof of concept, we propose an RL policy using Sparse-Sampling-based Monte Carlo Oracle and argue that it satisfies the stability property as long as the system dynamics under the optimal policy respects a Lyapunov function. The assumption of existence of a Lyapunov function is not restrictive as it

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshah2020stable

Related papers

• Learning To Stabilize Online Reinforcement Learning In Unbounded State Spaces (2023)0.00
• Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces (2024)2.26
• Sample Efficient Reinforcement Learning In Continuous State Spaces: A Perspective Beyond Linearity (2021)0.00
• Conservative Optimistic Policy Optimization Via Multiple Importance Sampling (2021)0.00
• Sublinear Regret For A Class Of Continuous-time Linear-quadratic Reinforcement Learning Problems (2024)0.00
• Policy Optimization Over General State And Action Spaces (2022)0.00
• Distributionally Robust Model-based Reinforcement Learning With Large State Spaces (2023)0.00
• Partially Observable RL With B-stability: Unified Structural Condition And Sharp Sample-efficient Algorithms (2022)0.00