Brittle AI, Causal Confusion, And Bad Mental Models: Challenges And Successes In The XAI Program

Abstract

The advances in artificial intelligence enabled by deep learning architectures are undeniable. In several cases, deep neural network driven models have surpassed human level performance in benchmark autonomy tasks. The underlying policies for these agents, however, are not easily interpretable. In fact, given their underlying deep models, it is impossible to directly understand the mapping from observations to actions for any reasonably complex agent. Producing this supporting technology to "open the black box" of these AI systems, while not sacrificing performance, was the fundamental goal of the DARPA XAI program. In our journey through this program, we have several "big picture" takeaways: 1) Explanations need to be highly tailored to their scenario; 2) many seemingly high performing RL agents are extremely brittle and are not amendable to explanation; 3) causal models allow for rich explanations, but how to present them isn't always straightforward; and 4) human subjects conjure fa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydruce2021brittle

Related papers

• Domain-level Explainability -- A Challenge For Creating Trust In Superhuman AI Strategies (2020)0.00
• Explainable Artificial Intelligence (XAI) For Increasing User Trust In Deep Reinforcement Learning Driven Autonomous Systems (2021)0.00
• Explainability In Deep Reinforcement Learning (2020)0.00
• Redefining Counterfactual Explanations For Reinforcement Learning: Overview, Challenges And Opportunities (2022)0.00
• A Survey Of Explainable Reinforcement Learning: Targets, Methods And Needs (2025)0.00
• MACIE: Multi-agent Causal Intelligence Explainer For Collective Behavior Understanding (2025)2.26
• Explainable Reinforcement Learning For Broad-xai: A Conceptual Framework And Survey (2021)0.00
• Explainable Deep Reinforcement Learning: State Of The Art And Challenges (2023)15.80