Agentic Entropy-balanced Policy Optimization

Abstract

Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydong2025agentic

Related papers

• Arbitrary Entropy Policy Optimization Breaks The Exploration Bottleneck Of Reinforcement Learning (2025)0.00
• EPO: Entropy-regularized Policy Optimization For LLM Agents Reinforcement Learning (2025)0.00
• EP-GRPO: Entropy-progress Aligned Group Relative Policy Optimization With Implicit Process Guidance (2026)0.00
• Examining Policy Entropy Of Reinforcement Learning Agents For Personalization Tasks (2022)0.00
• Predictable Reinforcement Learning Dynamics Through Entropy Rate Minimization (2023)0.00
• AEGPO: Adaptive Entropy-guided Policy Optimization For Diffusion Models (2026)0.00
• The Exploration-exploitation Dilemma Revisited: An Entropy Perspective (2024)0.00
• Understanding The Impact Of Entropy On Policy Optimization (2018)0.00