Human-level Control Through Directly-trained Deep Spiking Q-networks

Abstract

As the third-generation neural networks, Spiking Neural Networks (SNNs) have great potential on neuromorphic hardware because of their high energy-efficiency. However, Deep Spiking Reinforcement Learning (DSRL), i.e., the Reinforcement Learning (RL) based on SNNs, is still in its preliminary stage due to the binary output and the non-differentiable property of the spiking function. To address these issues, we propose a Deep Spiking Q-Network (DSQN) in this paper. Specifically, we propose a directly-trained deep spiking reinforcement learning architecture based on the Leaky Integrate-and-Fire (LIF) neurons and Deep Q-Network (DQN). Then, we adapt a direct spiking learning algorithm for the Deep Spiking Q-Network. We further demonstrate the advantages of using LIF neurons in DSQN theoretically. Comprehensive experiments have been conducted on 17 top-performing Atari games to compare our method with the state-of-the-art conversion method. The experimental results demonstrate the superiori

Stats

• citations44
• S2 citations—
• github stars0
• HF likes0
• heat score12.40
• arxiv keyliu2021human

Related papers

• Deep Reinforcement Learning With Spiking Q-learning (2022)0.00
• Fully Spiking Actor Network With Intra-layer Connections For Reinforcement Learning (2024)0.00
• Learning To Control Dynamical Agents Via Spiking Neural Networks And Metropolis-hastings Sampling (2025)0.00
• Distilling Neuron Spike With High Temperature In Reinforcement Learning Agents (2021)0.00
• Population-coding And Dynamic-neurons Improved Spiking Actor Network For Reinforcement Learning (2021)0.00
• Adaptive Surrogate Gradients For Sequential Reinforcement Learning In Spiking Neural Networks (2025)0.00
• A Low Latency Adaptive Coding Spiking Framework For Deep Reinforcement Learning (2022)7.16
• Hardware-aware Fine-tuning Of Spiking Q-networks On The Spinnaker2 Neuromorphic Platform (2025)0.00