Visualizing Dynamics: From T-sne To Semi-mdps

Abstract

Deep Reinforcement Learning (DRL) is a trending field of research, showing great promise in many challenging problems such as playing Atari, solving Go and controlling robots. While DRL agents perform well in practice we are still missing the tools to analayze their performance and visualize the temporal abstractions that they learn. In this paper, we present a novel method that automatically discovers an internal Semi Markov Decision Process (SMDP) model in the Deep Q Network's (DQN) learned representation. We suggest a novel visualization method that represents the SMDP model by a directed graph and visualize it above a t-SNE map. We show how can we interpret the agent's policy and give evidence for the hierarchical state aggregation that DQNs are learning automatically. Our algorithm is fully automatic, does not require any domain specific knowledge and is evaluated by a novel likelihood based evaluation criteria.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzrihem2016visualizing

Related papers

• Graying The Black Box: Understanding Dqns (2016)0.00
• Deepmdp: Learning Continuous Latent Space Models For Representation Learning (2019)0.00
• DQN With Model-based Exploration: Efficient Learning On Environments With Sparse Rewards (2019)0.00
• Interpretable Learning Dynamics In Unsupervised Reinforcement Learning (2025)0.00
• Deep Multi-agent Reinforcement Learning With Discrete-continuous Hybrid Action Spaces (2019)12.47
• Unsupervised Representation Learning In Deep Reinforcement Learning: A Review (2022)9.59
• Model-based Reinforcement Learning For Semi-markov Decision Processes With Neural Odes (2020)0.00
• Scalable Spectral Representations For Multi-agent Reinforcement Learning In Network Mdps (2024)0.00