Hrm-agent: Training A Recurrent Reasoning Model In Dynamic Environments Using Reinforcement Learning

Abstract

The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydang2025hrm

Related papers

• HTMRL: Biologically Plausible Reinforcement Learning With Hierarchical Temporal Memory (2020)0.00
• Exploring The Limits Of Hierarchical World Models In Reinforcement Learning (2024)6.34
• Modelling Bounded Rationality In Multi-agent Interactions By Generalized Recursive Reasoning (2019)9.23
• Soft Hierarchical Graph Recurrent Networks For Many-agent Partially Observable Environments (2021)0.00
• Probabilistic Recursive Reasoning For Multi-agent Reinforcement Learning (2019)0.00
• Hierarchical Reinforcement Learning In Complex 3D Environments (2023)0.00
• Explaining Agent's Decision-making In A Hierarchical Reinforcement Learning Scenario (2022)0.00
• Interactive Reinforcement Learning With Dynamic Reuse Of Prior Knowledge From Human/agent's Demonstration (2018)8.60