SMART: Shot-aware Multimodal Video Moment Retrieval With Audio-enhanced MLLM

Abstract

Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent progress in moment retrieval from videos using both traditional techniques and Multimodal Large Language Models (MLLM), most existing methods still rely on coarse temporal understanding and a single visual modality, limiting performance on complex videos. To address this, we introduce \textit\{S\}hot-aware \textit\{M\}ultimodal \textit\{A\}udio-enhanced \textit\{R\}etrieval of \textit\{T\}emporal \textit\{S\}egments (SMART), an MLLM-based framework that integrates audio cues and leverages shot-level temporal structure. SMART enriches multimodal representations by combining audio and visual features while applying \textbf\{Shot-aware Token Compression\}, which selectively retains high-information tokens within each shot to reduce redundancy and preserve fine-grained temporal details. We also refine prompt design

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyu2025smart

Related papers

• Query-centric Audio-visual Cognition Network For Moment Retrieval, Segmentation And Step-captioning (2024)3.58
• Language-based Audio Moment Retrieval (2024)3.58
• Temporal Working Memory: Query-guided Segment Refinement For Enhanced Multimodal Understanding (2025)11.33
• Videollama 2: Advancing Spatial-temporal Modeling And Audio Understanding In Video-llms (2024)0.00
• WAVE: Learning Unified & Versatile Audio-visual Embeddings With Multimodal LLM (2025)0.00
• Realizing Video Summarization From The Path Of Language-based Semantic Understanding (2024)0.00
• SSVMR: Saliency-based Self-training For Video-music Retrieval (2023)8.09
• Semantically Consistent Video-to-audio Generation Using Multimodal Language Large Model (2024)0.00