Toward Interpretable Deep Reinforcement Learning With Linear Model U-trees

Abstract

Deep Reinforcement Learning (DRL) has achieved impressive success in many applications. A key component of many DRL models is a neural network representing a Q function, to estimate the expected cumulative reward following a state-action pair. The Q function neural network contains a lot of implicit knowledge about the RL problems, but often remains unexamined and uninterpreted. To our knowledge, this work develops the first mimic learning framework for Q functions in DRL. We introduce Linear Model U-trees (LMUTs) to approximate neural network predictions. An LMUT is learned using a novel on-line algorithm that is well-suited for an active play setting, where the mimic learner observes an ongoing interaction between the neural net and the environment. Empirical evaluation shows that an LMUT mimics a Q function substantially better than five baseline methods. The transparent tree structure of an LMUT facilitates understanding the network's learned knowledge by analyzing feature influenc

Stats

• citations54
• S2 citations—
• github stars0
• HF likes0
• heat score13.05
• arxiv keyliu2018toward

Related papers

• Upside-down Reinforcement Learning For More Interpretable Optimal Control (2024)0.00
• CDT: Cascading Decision Trees For Explainable Reinforcement Learning (2020)0.00
• Mitigating Information Loss In Tree-based Reinforcement Learning Via Direct Optimization (2024)0.00
• Temporal Difference Models: Model-free Deep RL For Model-based Control (2018)0.00
• DQN-TAMER: Human-in-the-loop Reinforcement Learning With Intractable Feedback (2018)0.00
• Using Monte Carlo Tree Search As A Demonstrator Within Asynchronous Deep RL (2018)0.00
• Optimizing Interpretable Decision Tree Policies For Reinforcement Learning (2024)0.00
• Simplifying Model-based RL: Learning Representations, Latent-space Models, And Policies With One Objective (2022)0.00