Robust Task Representations For Offline Meta-reinforcement Learning Via Contrastive Learning
2022 · Haoqi Yuan, Zongqing Lu
Abstract
We study offline meta-reinforcement learning, a practical reinforcement learning paradigm that learns from offline data to adapt to new tasks. The distribution of offline data is determined jointly by the behavior policy and the task. Existing offline meta-reinforcement learning algorithms cannot distinguish these factors, making task representations unstable to the change of behavior policies. To address this problem, we propose a contrastive learning framework for task representations that are robust to the distribution mismatch of behavior policies in training and test. We design a bi-level encoder structure, use mutual information maximization to formalize task representation learning, derive a contrastive learning objective, and introduce several approaches to approximate the true distribution of negative pairs. Experiments on a variety of offline meta-reinforcement learning benchmarks demonstrate the advantages of our method over prior methods, especially on the generalization to
Authors
(none)
Tags
Stats
Related papers
- Entropy Regularized Task Representation Learning For Offline Meta-reinforcement Learning (2024)0.00
- Provably Improved Context-based Offline Meta-rl With Attention And Contrastive Learning (2021)0.00
- FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization (2020)0.00
- On Context Distribution Shift In Task Representation Learning For Offline Meta RL (2023)0.00
- Offline Meta-reinforcement Learning With Online Self-supervision (2021)0.00
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-based Offline Meta Reinforcement Learning (2024)0.00
- Contrastive UCB: Provably Efficient Contrastive Self-supervised Learning In Online Reinforcement Learning (2022)0.00
- Offline Meta-reinforcement Learning With Advantage Weighting (2020)0.00