On Instrumental Variable Regression For Deep Offline Policy Evaluation

Abstract

We show that the popular reinforcement learning (RL) strategy of estimating the state-action value (Q-function) by minimizing the mean squared Bellman error leads to a regression problem with confounding, the inputs and output noise being correlated. Hence, direct minimization of the Bellman error can result in significantly biased Q-function estimates. We explain why fixing the target Q-network in Deep Q-Networks and Fitted Q Evaluation provides a way of overcoming this confounding, thus shedding new light on this popular but not well understood trick in the deep RL literature. An alternative approach to address confounding is to leverage techniques developed in the causality literature, notably instrumental variables (IV). We bring together here the literature on IV and RL by investigating whether IV approaches can lead to improved Q-function estimates. This paper analyzes and compares a wide range of recent IV methods in the context of offline policy evaluation (OPE), where the goal

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2021on

Related papers

• Instrumental Variable Value Iteration For Causal Offline Reinforcement Learning (2021)0.00
• Offline Reinforcement Learning With Instrumental Variables In Confounded Markov Decision Processes (2022)0.00
• Offline RL With No OOD Actions: In-sample Learning Via Implicit Value Regularization (2023)0.00
• Offline Policy Evaluation For Reinforcement Learning With Adaptively Collected Data (2023)0.00
• A Complete Characterization Of Linear Estimators For Offline Policy Evaluation (2022)0.00
• A Perspective Of Q-value Estimation On Offline-to-online Reinforcement Learning (2023)7.81
• Near-optimal Provable Uniform Convergence In Offline Policy Evaluation For Reinforcement Learning (2020)0.00
• Counterfactual-augmented Importance Sampling For Semi-offline Policy Evaluation (2023)0.00