Automaton Distillation: Neuro-symbolic Transfer Learning For Deep Reinforcement Learning

Abstract

Reinforcement learning (RL) is a powerful tool for finding optimal policies in sequential decision processes. However, deep RL methods have two weaknesses: collecting the amount of agent experience required for practical RL problems is prohibitively expensive, and the learned policies exhibit poor generalization on tasks outside the training data distribution. To mitigate these issues, we introduce automaton distillation, a form of neuro-symbolic transfer learning in which Q-value estimates from a teacher are distilled into a low-dimensional representation in the form of an automaton. We then propose methods for generating Q-value estimates where symbolic information is extracted from a teacher's Deep Q-Network (DQN). The resulting Q-value estimates are used to bootstrap learning in the target discrete and continuous environment via a modified DQN and Twin-Delayed Deep Deterministic (TD3) loss function, respectively. We demonstrate that automaton distillation decreases the time require

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysingireddy2023automaton

Related papers

• Efficient Open-world Reinforcement Learning Via Knowledge Distillation And Autonomous Rule Discovery (2023)0.00
• On The Transferability Of Deep-q Networks (2021)0.00
• Adaptive \(q\)-network: On-the-fly Target Selection For Deep Reinforcement Learning (2024)0.00
• Goal-conditioned Q-learning As Knowledge Distillation (2022)3.58
• Continual Policy Distillation From Distributed Reinforcement Learning Teachers (2026)0.00
• Low Precision Policy Distillation With Application To Low-power, Real-time Sensation-cognition-action Loop With Neuromorphic Computing (2018)0.00
• Systematic Generalisation Through Task Temporal Logic And Deep Reinforcement Learning (2020)0.00
• Learning Symbolic Representations For Reinforcement Learning Of Non-markovian Behavior (2023)0.00