Task-adaptive Retrieval Over Agentic Multi-modal Web Histories Via Learned Graph Memory

Abstract

Retrieving relevant observations from long multi-modal web interaction histories is challenging because relevance depends on the evolving task state, modality (screenshots, HTML text, structured signals), and temporal distance. Prior approaches typically rely on static similarity thresholds or fixed-capacity buffers, which fail to adapt relevance to the current task context. We propose \textbf\{ACGM\}, a learned graph-memory retriever that constructs *task-adaptive* relevance graphs over agent histories using policy-gradient optimization from downstream task success. ACGM captures heterogeneous temporal dynamics with modality-specific decay (visual decays \(4.3\times\) faster than text: \(\lambda_v\{=\}0.47\) vs.\ \(\lambda_x\{=\}0.11\)) and learns sparse connectivity (3.2 edges/node), enabling efficient \(O(log T)\) retrieval. Across WebShop, VisualWebArena, and Mind2Web, ACGM improves retrieval quality to \textbf\{82.7 nDCG@10\} (+9.3 over GPT-4o, \(p\{<\}0.001\)) and \textbf\{89.2%

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyforouzandeh2026task

Related papers

• GAAMA: Graph Augmented Associative Memory For Agents (2026)0.00
• MG\(^2\)-RAG: Multi-granularity Graph For Multimodal Retrieval-augmented Generation (2026)0.00
• Multimodal RAG For Unstructured Data:leveraging Modality-aware Knowledge Graphs With Hybrid Retrieval (2025)0.00
• Optimizing Recall Or Relevance? A Multi-task Multi-head Approach For Item-to-item Retrieval In Recommendation (2025)0.00
• Gpu-accelerated Multi-relational Parallel Graph Retrieval For Web-scale Recommendations (2025)0.00
• Graph-based Hierarchical Relevance Matching Signals For Ad-hoc Retrieval (2021)8.60
• A Dynamic Retrieval-augmented Generation System With Selective Memory And Remembrance (2026)0.00
• Attention Grounded Enhancement For Visual Document Retrieval (2025)0.00