On The Continuity And Smoothness Of The Value Function In Reinforcement Learning And Optimal Control

Abstract

The value function plays a crucial role as a measure for the cumulative future reward an agent receives in both reinforcement learning and optimal control. It is therefore of interest to study how similar the values of neighboring states are, i.e., to investigate the continuity of the value function. We do so by providing and verifying upper bounds on the value function's modulus of continuity. Additionally, we show that the value function is always H\"older continuous under relatively weak assumptions on the underlying system and that non-differentiable value functions can be made differentiable by slightly "disturbing" the system.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyharder2024on

Related papers

• Convex Programs And Lyapunov Functions For Reinforcement Learning: A Unified Perspective On The Analysis Of Value-based Methods (2022)2.26
• Continuous-time Value Function Approximation In Reproducing Kernel Hilbert Spaces (2018)0.00
• Statistical Inference Of The Value Function For Reinforcement Learning In Infinite Horizon Settings (2020)13.14
• Deep Radial-basis Value Functions For Continuous Control (2020)0.00
• On Value Functions And The Agent-environment Boundary (2019)0.00
• Continuous-time Reinforcement Learning: Ellipticity Enables Model-free Value Function Approximation (2026)0.00
• On The Limited Representational Power Of Value Functions And Its Links To Statistical (in)efficiency (2024)0.00
• Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces (2024)2.26