Robust Reinforcement Learning: A Case Study In Linear Quadratic Regulation

Abstract

This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, we revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? Using advanced stability results in control theory, it is shown that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded, and, moreover, enter and stay in a small neighbourhood of the optimal LQR solution. As an application, a novel off-policy optimistic least-squares policy iteration for the LQR problem is proposed, when the system dynamics are subjected to additive stochastic disturbances. The proposed new results in robust rei

Stats

• citations30
• S2 citations—
• github stars0
• HF likes0
• heat score11.19
• arxiv keypang2020robust

Related papers

• Learning Robust Control For LQR Systems With Multiplicative Noise Via Policy Gradient (2019)0.00
• Least-squares Temporal Difference Learning For The Linear Quadratic Regulator (2017)0.00
• Finite-time Analysis Of Approximate Policy Iteration For The Linear Quadratic Regulator (2019)0.00
• Sample Complexity Of The Linear Quadratic Regulator: A Reinforcement Learning Lens (2024)0.00
• On The Optimization Landscape Of Dynamic Output Feedback: A Case Study For Linear Quadratic Regulator (2022)4.52
• Enhancing Robustness In Deep Reinforcement Learning: A Lyapunov Exponent Approach (2024)0.00
• Learning The Linear Quadratic Regulator From Nonlinear Observations (2020)0.00
• Alternating Optimisation And Quadrature For Robust Control (2016)7.16