STEERING: Stein Information Directed Exploration For Model-based Reinforcement Learning

Abstract

Directed Exploration is a crucial challenge in reinforcement learning (RL), especially when rewards are sparse. Information-directed sampling (IDS), which optimizes the information ratio, seeks to do so by augmenting regret with information gain. However, estimating information gain is computationally intractable or relies on restrictive assumptions which prohibit its use in many practical instances. In this work, we posit an alternative exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal, which under suitable conditions, can be computed in closed form with the kernelized Stein discrepancy (KSD). Based on KSD, we develop a novel algorithm \algo: \textbf\{STE\}in information dir\textbf\{E\}cted exploration for model-based \textbf\{R\}einforcement Learn\textbf\{ING\}. To enable its derivation, we develop fundamentally new variants of KSD for discrete conditional distributions. \{We further esta

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychakraborty2023steering

Related papers

• Information-directed Exploration For Deep Reinforcement Learning (2018)0.00
• Information Directed Reward Learning For Reinforcement Learning (2021)0.00
• DEIR: Efficient And Robust Exploration Through Discriminative-model-based Episodic Intrinsic Rewards (2023)0.00
• Maxinforl: Boosting Exploration In Reinforcement Learning Through Information Gain Maximization (2024)0.00
• Exploration And Incentives In Reinforcement Learning (2021)8.09
• Information Content Exploration (2023)0.00
• Deterministic Sequencing Of Exploration And Exploitation For Reinforcement Learning (2022)5.84
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00