Deep Inverse Q-learning With Constraints

Abstract

Popular Maximum Entropy Inverse Reinforcement Learning approaches require the computation of expected state visitation frequencies for the optimal policy under an estimate of the reward function. This usually requires intermediate value estimation in the inner loop of the algorithm, slowing down convergence considerably. In this work, we introduce a novel class of algorithms that only needs to solve the MDP underlying the demonstrated behavior once to recover the expert policy. This is possible through a formulation that exploits a probabilistic behavior assumption for the demonstrations within the structure of Q-learning. We propose Inverse Action-value Iteration which is able to fully recover an underlying reward of an external agent in closed-form analytically. We further provide an accompanying class of sampling-based variants which do not depend on a model of the environment. We show how to extend this class of algorithms to continuous state-spaces via function approximation and h

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykalweit2020deep

Related papers

• Handling Cost And Constraints With Off-policy Deep Reinforcement Learning (2023)0.00
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• Maximum-likelihood Inverse Reinforcement Learning With Finite-time Guarantees (2022)0.00
• Statistical Analysis Of Inverse Entropy-regularized Reinforcement Learning (2025)0.00
• Inverse Reinforcement Learning With Explicit Policy Estimates (2021)2.26
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Learning Safe, Constrained Policies Via Imitation Learning: Connection To Probabilistic Inference And A Naive Algorithm (2025)0.00
• Maximum-entropy Exploration With Future State-action Visitation Measures (2026)0.00