Hil-bench (human-in-loop Benchmark): Do Agents Know When To Ask For Help?

Abstract

arXiv:2604.09408v4 Announce Type: replace Abstract: Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytrinh2026hil

Related papers

• Inquiremobile: Teaching Vlm-based Mobile Agent To Request Human Assistance Via Reinforcement Fine-tuning (2026)0.00
• Reward Hacking Benchmark: Measuring Exploits In LLM Agents With Tool Use (2026)0.00
• The Agent Capability Problem: Predicting Solvability Through Information-theoretic Bounds (2025)0.00
• Aligning Agents Via Planning: A Benchmark For Trajectory-level Reward Modeling (2026)0.00
• Ask-ac: An Initiative Advisor-in-the-loop Actor-critic Framework (2022)3.58
• Human Engagement Providing Evaluative And Informative Advice For Interactive Reinforcement Learning (2020)9.23
• A Bayesian Solution To The Imitation Gap (2024)0.00
• When Your Ais Deceive You: Challenges Of Partial Observability In Reinforcement Learning From Human Feedback (2024)0.00