Softdice For Imitation Learning: Rethinking Off-policy Distribution Matching
2021 · Mingfei Sun, Anuj Mahajan, Katja Hofmann, et al.
Abstract
We present SoftDICE, which achieves state-of-the-art performance for imitation learning. SoftDICE fixes several key problems in ValueDICE, an off-policy distribution matching approach for sample-efficient imitation learning. Specifically, the objective of ValueDICE contains logarithms and exponentials of expectations, for which the mini-batch gradient estimate is always biased. Second, ValueDICE regularizes the objective with replay buffer samples when expert demonstrations are limited in number, which however changes the original distribution matching problem. Third, the re-parametrization trick used to derive the off-policy objective relies on an implicit assumption that rarely holds in training. We leverage a novel formulation of distribution matching and consider an entropy-regularized off-policy objective, which yields a completely offline algorithm called SoftDICE. Our empirical results show that SoftDICE recovers the expert policy with only one demonstration trajectory and no fu
Authors
(none)
Tags
Stats
Related papers
- Offline Imitation Learning With Suboptimal Demonstrations Via Relaxed Distribution Matching (2023)6.77
- Diffusion-dice: In-sample Diffusion Guidance For Offline Reinforcement Learning (2024)0.00
- Dualdice: Behavior-agnostic Estimation Of Discounted Stationary Distribution Corrections (2019)0.00
- Deterministic And Discriminative Imitation (d2-imitation): Revisiting Adversarial Imitation For Sample Efficiency (2021)0.00
- Distributional Soft Actor-critic: Off-policy Reinforcement Learning For Addressing Value Estimation Errors (2020)17.77
- Strictly Batch Imitation Learning By Energy-based Distribution Matching (2020)0.00
- ODICE: Revealing The Mystery Of Distribution Correction Estimation Via Orthogonal-gradient Update (2024)0.00
- Lobsdice: Offline Learning From Observation Via Stationary Distribution Correction Estimation (2022)0.00