Sample-efficient Online Learning In LM Agents Via Hindsight Trajectory Rewriting

Abstract

Language model (LM) agents deployed in novel environments often exhibit poor sample efficiency when learning from sequential interactions. This significantly hinders the usefulness of such agents in environments where interaction is costly (for example, when they interact with humans or reset physical systems). While a number of existing LM agent architectures incorporate various mechanisms for experience storage and reflection, they make limited use of LMs' abilities to directly generate or reason about full counterfactual trajectories. We introduce ECHO (Experience Consolidation via Hindsight Optimization), a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents. ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts, effectively creating synthetic positive examples from unsuccessful interactions. Our approach consists of two components: a hindsight rule that uses the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score1.57
• arxiv keyhu2025sample

Related papers

• Agenther: Hindsight Experience Replay For LLM Agent Trajectory Relabeling (2026)0.00
• SAC-GLAM: Improving Online RL For LLM Agents With Soft Actor-critic And Hindsight Relabeling (2024)0.00
• Higher : Improving Instruction Following With Hindsight Generation For Experience Replay (2019)6.34
• Training Agents With Weakly Supervised Feedback From Large Language Models (2024)0.00
• Himac: Hierarchical Macro-micro Learning For Long-horizon LLM Agents (2026)0.00
• Experiential Reinforcement Learning (2026)2.58
• Spark: Strategic Policy-aware Exploration Via Dynamic Branching For Long-horizon Agentic Learning (2026)0.00
• Agent-pro: Learning To Evolve Via Policy-level Reflection And Optimization (2024)9.59