Optimization Solution Functions As Deterministic Policies For Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykhattar2024optimization

Related papers

• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• Optimal Conservative Offline RL With General Function Approximation Via Augmented Lagrangian (2022)0.00
• POPO: Pessimistic Offline Policy Optimization (2020)5.24
• Adversarially Trained Weighted Actor-critic For Safe Offline Reinforcement Learning (2024)0.00
• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Decision-aware Actor-critic With Function Approximation And Theoretical Guarantees (2023)0.00
• Provable Benefits Of Actor-critic Methods For Offline Reinforcement Learning (2021)0.00