Abstract
User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models due to the substantial gap between high-level user instructions and low-level agent actions. While recent advancements in language models (e.g., LLMs, VLMs, VLAs) have improved instruction comprehension, these models face critical limitations in either inference speed (LLMs, VLMs) or generalizability (VLAs) for UC-EVT tasks. To address these challenges, we propose \textbf\{Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT)\} agent, which bridges instruction comprehension and action generation using \textit\{spatial goals\} as intermediaries. HIEVT first introduces \textit\{LLM-based Semantic-Spatial Goal Aligner\} to translate diverse human instructions into spatial goals that directly annotate the desired spatial position. Then the \textit\{RL-based Adaptive Goal-Aligned Policy\}, a general offline policy, enables the tracker to position the target as specifie