Anytime-valid Off-policy Inference For Contextual Bandits

Abstract

Contextual bandit algorithms are ubiquitous tools for active sequential experimentation in healthcare and the tech industry. They involve online learning algorithms that adaptively learn policies over time to map observed contexts \(X_t\) to actions \(A_t\) in an attempt to maximize stochastic rewards \(R_t\). This adaptivity raises interesting but hard statistical inference questions, especially counterfactual ones: for example, it is often of interest to estimate the properties of a hypothetical policy that is different from the logging policy that was used to collect the data -- a problem known as ``off-policy evaluation'' (OPE). Using modern martingale techniques, we present a comprehensive framework for OPE inference that relax unnecessary conditions made in some past works, significantly improving on them both theoretically and empirically. Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post-hoc), when the logging

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keywaudbysmith2022anytime

Related papers

• Local Metric Learning For Off-policy Evaluation In Contextual Bandits With Continuous Actions (2022)0.00
• Off-policy Evaluation And Learning From Logged Bandit Feedback: Error Reduction Via Surrogate Policy (2018)0.00
• Confidence Interval For Off-policy Evaluation From Dependent Samples Via Bandit Algorithm: Approach From Standardized Martingales (2020)0.00
• A General Framework For Off-policy Learning With Partially-observed Reward (2025)0.00
• A New Bandit Setting Balancing Information From State Evolution And Corrupted Context (2020)0.00
• Contextual Bandits And Optimistically Universal Learning (2022)0.00
• Inverse Contextual Bandits Without Rewards: Learning From A Non-stationary Learner Via Suffix Imitation (2026)0.00
• Intrinsically Efficient, Stable, And Bounded Off-policy Evaluation For Reinforcement Learning (2019)0.00