A Minimax Learning Approach To Off-policy Evaluation In Confounded Partially Observable Markov Decision Processes

Abstract

We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes (POMDPs), where the evaluation policy depends only on observable variables and the behavior policy depends on unobservable latent variables. Existing works either assume no unmeasured confounders, or focus on settings where both the observation and the state spaces are tabular. In this work, we first propose novel identification methods for OPE in POMDPs with latent confounders, by introducing bridge functions that link the target policy's value and the observed data distribution. We next propose minimax estimation methods for learning these bridge functions, and construct three estimators based on these estimated bridge functions, corresponding to a value function-based estimator, a marginalized importance sampling estimator, and a doubly-robust estimator. Our proposal permits general function approximation and is thus applicable to settings with continuous or large observation/state spaces. The

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshi2021a

Related papers

• A Spectral Approach To Off-policy Evaluation For Pomdps (2021)0.00
• Future-dependent Value-based Off-policy Evaluation In Pomdps (2022)0.00
• Off-policy Evaluation In Infinite-horizon Reinforcement Learning With Latent Confounders (2020)0.00
• An Instrumental Variable Approach To Confounded Off-policy Evaluation (2022)0.00
• Statistical Tractability Of Off-policy Evaluation Of History-dependent Policies In Pomdps (2025)0.00
• Proximal Reinforcement Learning: Efficient Off-policy Evaluation In Partially Observed Markov Decision Processes (2021)0.00
• A Maximum-entropy Approach To Off-policy Evaluation In Average-reward Mdps (2020)0.00
• A Policy Gradient Method For Confounded Pomdps (2023)0.00