Context-enhanced Video Moment Retrieval With Large Language Models

Abstract

Current methods for Video Moment Retrieval (VMR) struggle to align complex situations involving specific environmental details, character descriptions, and action narratives. To tackle this issue, we propose a Large Language Model-guided Moment Retrieval (LMR) approach that employs the extensive knowledge of Large Language Models (LLMs) to improve video context representation as well as cross-modal alignment, facilitating accurate localization of target moments. Specifically, LMR introduces a context enhancement technique with LLMs to generate crucial target-related context semantics. These semantics are integrated with visual features for producing discriminative video representations. Finally, a language-conditioned transformer is designed to decode free-form language queries, on the fly, using aligned video representations for moment retrieval. Extensive experiments demonstrate that LMR achieves state-of-the-art results, outperforming the nearest competitor by up to 3.28% and 4.06%

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyliu2024context

Related papers

• Prior Knowledge Integration Via LLM Encoding And Pseudo Event Regulation For Video Moment Retrieval (2024)13.83
• Vlanet: Video-language Alignment Network For Weakly-supervised Video Moment Retrieval (2020)13.28
• Towards Balanced Alignment: Modal-enhanced Semantic Modeling For Video Moment Retrieval (2023)14.33
• Hybrid-learning Video Moment Retrieval Across Multi-domain Labels (2024)0.00
• Vill-e: Video LLM Embeddings For Retrieval (2026)0.00
• Verve: Versatile Retrieval For Videos Via Unified Embeddings (2026)0.00
• Vidvec: Unlocking Video MLLM Embeddings For Video-text Retrieval (2026)0.00
• MERLIN: Multimodal Embedding Refinement Via Llm-based Iterative Navigation For Text-video Retrieval-rerank Pipeline (2024)5.84