Optimal Conservative Offline RL With General Function Approximation Via Augmented Lagrangian

Abstract

Offline reinforcement learning (RL), which refers to decision-making from a previously-collected dataset of interactions, has received significant attention over the past years. Much effort has focused on improving offline RL practicality by addressing the prevalent issue of partial data coverage through various forms of conservative policy learning. While the majority of algorithms do not have finite-sample guarantees, several provable conservative offline RL algorithms are designed and analyzed within the single-policy concentrability framework that handles partial coverage. Yet, in the nonlinear function approximation setting where confidence intervals are difficult to obtain, existing provable algorithms suffer from computational intractability, prohibitively strong assumptions, and suboptimal statistical rates. In this paper, we leverage the marginalized importance sampling (MIS) formulation of RL and present the first set of offline RL algorithms that are statistically optimal an

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyrashidinejad2022optimal

Related papers

• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Minimax Optimal And Computationally Efficient Algorithms For Distributionally Robust Offline Reinforcement Learning (2024)0.00
• What Are The Statistical Limits Of Offline RL With Linear Function Approximation? (2020)0.00
• Provably Efficient Offline Goal-conditioned Reinforcement Learning With General Function Approximation And Single-policy Concentrability (2023)0.00
• Nearly Minimax Optimal Offline Reinforcement Learning With Linear Function Approximation: Single-agent MDP And Markov Game (2022)0.00
• Neural Network Approximation For Pessimistic Offline Reinforcement Learning (2023)0.00
• Sample Efficient Active Algorithms For Offline Reinforcement Learning (2026)0.00
• Offline Reinforcement Learning: Fundamental Barriers For Value Function Approximation (2021)0.00