Dual RL: Unification And New Methods For Reinforcement And Imitation Learning

Abstract

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL m

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysikchi2023dual

Related papers

• A Policy-guided Imitation Approach For Offline Reinforcement Learning (2022)0.00
• Bridging Offline Reinforcement Learning And Imitation Learning: A Tale Of Pessimism (2021)0.00
• Curriculum Offline Imitation Learning (2021)0.00
• Blending Imitation And Reinforcement Learning For Robust Policy Improvement (2023)0.00
• A Primal-dual Algorithm For Offline Constrained Reinforcement Learning With Linear Mdps (2024)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Offline Reinforcement Learning With Imbalanced Datasets (2023)0.00
• RLIF: Interactive Imitation Learning As Reinforcement Learning (2023)0.00