BRAC+: Improved Behavior Regularized Actor Critic For Offline Reinforcement Learning

Abstract

Online interactions with the environment to collect data samples for training a Reinforcement Learning (RL) agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning is to address this problem by learning effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out-of-distribution (less explored) actions and are hence unsuitable for Offline RL. Behavior regularization, which constraints the learned policy within the support set of the dataset, has been proposed to tackle the limitations of standard off-policy algorithms. In this paper, we improve the behavior regularized offline reinforcement learning and propose BRAC+. First, we propose quantification of the out-of-distribution actions and conduct comparisons between using Kullback-Leibler divergence versus using Maximum Mean Discrepancy as the regularization protocol. We propose an analytical upper bound o

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2021brac

Related papers

• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• Iteratively Refined Behavior Regularization For Offline Reinforcement Learning (2023)2.26
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• B3C: A Minimalist Approach To Offline Multi-agent Reinforcement Learning (2025)0.00
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• TD3 With Reverse KL Regularizer For Offline Reinforcement Learning From Mixed Datasets (2022)4.52
• FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization (2020)0.00
• Offline Reinforcement Learning With Fisher Divergence Critic Regularization (2021)0.00