Provable Benefits Of Actor-critic Methods For Offline Reinforcement Learning

Abstract

Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle, leading to several key advantages compared to the state of the art. The algorithm can operate when the Bellman evaluation operator is closed with respect to the action value function of the actor's policies; this is a more general setting than the low-rank MDP model. Despite the added generality, the procedure is computationally tractable as it involves the solution of a sequence of second-order programs. We prove an upper bound on the suboptimality gap of the policy returned by the procedure that depends on the data coverage of any arbitrary, possibly data dependent comparator policy. The achievable guarantee is complemented with a minimax lower bound that is matching up to logarithmic factors.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzanette2021provable

Related papers

• Importance Weighted Actor-critic For Optimal Conservative Offline Reinforcement Learning (2023)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Bellman-consistent Pessimism For Offline Reinforcement Learning (2021)0.00
• Optimization Solution Functions As Deterministic Policies For Offline Reinforcement Learning (2024)0.00
• Doubly Robust Off-policy Actor-critic Algorithms For Reinforcement Learning (2019)0.00
• Relu To The Rescue: Improve Your On-policy Actor-critic With Positive Advantages (2023)0.00
• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• Decision-aware Actor-critic With Function Approximation And Theoretical Guarantees (2023)0.00