FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization

Abstract

We study the offline meta-reinforcement learning (OMRL) problem, a paradigm which enables reinforcement learning (RL) algorithms to quickly adapt to unseen tasks without any interactions with the environments, making RL truly practical in many real-world applications. This problem is still not fully understood, for which two major challenges need to be addressed. First, offline RL usually suffers from bootstrapping errors of out-of-distribution state-actions which leads to divergence of value functions. Second, meta-RL requires efficient and robust task inference learned jointly with control policy. In this work, we enforce behavior regularization on learned policy as a general approach to offline RL, combined with a deterministic context encoder for efficient task inference. We propose a novel negative-power distance metric on bounded context embedding space, whose gradients propagation is detached from the Bellman backup. We provide analysis and insight showing that some simple desig

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2020focal

Related papers

• Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-based Offline Meta Reinforcement Learning (2024)0.00
• Provably Improved Context-based Offline Meta-rl With Attention And Contrastive Learning (2021)0.00
• Entropy Regularized Task Representation Learning For Offline Meta-reinforcement Learning (2024)0.00
• Offline Meta Learning Of Exploration (2020)0.00
• On Context Distribution Shift In Task Representation Learning For Offline Meta RL (2023)0.00
• Goal-conditioned Reinforcement Learning From Sub-optimal Data On Metric Spaces (2024)0.00
• BRAC+: Improved Behavior Regularized Actor Critic For Offline Reinforcement Learning (2021)0.00
• Robust Task Representations For Offline Meta-reinforcement Learning Via Contrastive Learning (2022)0.00