The Termination Critic

Abstract

In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We propose an algorithm that focuses on the termination condition, as opposed to -- as is common -- the policy. The termination condition is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option's encoding -- arguably a key reason for using abstractions. To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a "critic" for the termination condition. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively meaningful, and useful for learning and planning.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyharutyunyan2019the

Related papers

• Attention Option-critic (2022)0.00
• Performance Dynamics And Termination Errors In Reinforcement Learning: A Unifying Perspective (2019)5.84
• Autonomous Option Invention For Continual Hierarchical Reinforcement Learning And Planning (2024)2.26
• A Theory Of Abstraction In Reinforcement Learning (2022)0.00
• Tackling Uncertainties In Multi-agent Reinforcement Learning Through Integration Of Agent Termination Dynamics (2025)2.26
• Option-critic In Cooperative Multi-agent Systems (2019)0.00
• A Provably Efficient Option-based Algorithm For Both High-level And Low-level Learning (2024)0.00
• Curiosity Killed Or Incapacitated The Cat And The Asymptotically Optimal Agent (2020)0.00