Viper: Provably Efficient Algorithm For Offline RL With Neural Function Approximation

Abstract

We propose a novel algorithm for offline reinforcement learning called Value Iteration with Perturbed Rewards (VIPeR), which amalgamates the pessimism principle with random perturbations of the value function. Most current offline RL algorithms explicitly construct statistical confidence regions to obtain pessimism via lower confidence bounds (LCB), which cannot easily scale to complex problems where a neural network is used to estimate the value functions. Instead, VIPeR implicitly obtains pessimism by simply perturbing the offline data multiple times with carefully-designed i.i.d. Gaussian noises to learn an ensemble of estimated state-action \{value functions\} and acting greedily with respect to the minimum of the ensemble. The estimated state-action values are obtained by fitting a parametric model (e.g., neural networks) to the perturbed datasets using gradient descent. As a result, VIPeR only needs \(\mathcal\{O\}(1)\) time complexity for action selection, while LCB-based algori

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynguyentang2023viper

Related papers

• Pessimistic Nonlinear Least-squares Value Iteration For Offline Reinforcement Learning (2023)0.00
• Is Pessimism Provably Efficient For Offline RL? (2020)0.00
• Neural Network Approximation For Pessimistic Offline Reinforcement Learning (2023)0.00
• Diverse Randomized Value Functions: A Provably Pessimistic Approach For Offline Reinforcement Learning (2024)3.58
• Near-optimal Offline Reinforcement Learning With Linear Representation: Leveraging Variance Information With Pessimism (2022)0.00
• Optimal Conservative Offline RL With General Function Approximation Via Augmented Lagrangian (2022)0.00
• Offline Reinforcement Learning: Fundamental Barriers For Value Function Approximation (2021)0.00
• Sample Efficient Active Algorithms For Offline Reinforcement Learning (2026)0.00