Knowledge Completes The Vision: A Multimodal Entity-aware Retrieval-augmented Generation Framework For News Image Captioning

Abstract

News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyou2025knowledge

Related papers

• Towards Retrieval-augmented Architectures For Image Captioning (2024)9.41
• Event-retriever: Event-aware Multimodal Image Retrieval For Realistic Captions (2025)0.00
• Recap: Event-aware Image Captioning With Article Retrieval And Semantic Gaussian Normalization (2025)1.56
• Cross-modal Retrieval For Knowledge-based Visual Question Answering (2024)7.81
• Show, Translate And Tell (2019)4.52
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• Retrieval-augmented Image Captioning (2023)11.29
• Retrieval-augmented Multimodal Language Modeling (2022)0.00