Retrieval-augmented Egocentric Video Captioning

Abstract

Understanding human actions from videos of first-person view poses significant challenges. Most prior approaches explore representation learning on egocentric videos only, while overlooking the potential benefit of exploiting existing large-scale third-person videos. In this paper, (1) we develop EgoInstructor, a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos to enhance the video captioning of egocentric videos. (2) For training the cross-view retrieval module, we devise an automatic pipeline to discover ego-exo video pairs from distinct large-scale egocentric and exocentric datasets. (3) We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions. (4) Through extensive experiments, our cross-view retrieval module demonstrates superior performance across seven benc

Stats

• citations31
• S2 citations—
• github stars0
• HF likes0
• heat score11.29
• arxiv keyxu2024retrieval

Related papers

• From Third Person To First Person: Dataset And Baselines For Synthesis And Retrieval (2018)0.00
• Helping Hands: An Object-aware Ego-centric Video Recognition Model (2023)10.07
• Egocvr: An Egocentric Benchmark For Fine-grained Composed Video Retrieval (2024)10.00
• Fine-grained Action Retrieval Through Multiple Parts-of-speech Embeddings (2019)15.62
• An Overview Of Challenges In Egocentric Text-video Retrieval (2023)0.00
• Narrating The Video: Boosting Text-video Retrieval Via Comprehensive Utilization Of Frame-level Captions (2025)6.77
• Retrieval-augmented Image Captioning (2023)11.29
• Egocentric Video-language Pretraining @ EPIC-KITCHENS-100 Multi-instance Retrieval Challenge 2022 (2022)4.83