Towards Retrieval-augmented Architectures For Image Captioning

Abstract

The objective of image captioning models is to bridge the gap between the visual and linguistic modalities by generating natural language descriptions that accurately reflect the content of input images. In recent years, researchers have leveraged deep learning-based models and made advances in the extraction of visual features and the design of multimodal connections to tackle this task. This work presents a novel approach towards developing image captioning models that utilize an external kNN memory to improve the generation process. Specifically, we propose two model variants that incorporate a knowledge retriever component that is based on visual similarities, a differentiable encoder to represent input images, and a kNN-augmented language model to predict tokens based on contextual cues and text retrieved from the external memory. We experimentally validate our approach on COCO and nocaps datasets and demonstrate that incorporating an explicit external memory can significantly enh

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keysarto2024towards

Related papers

• Retrieval-augmented Image Captioning (2023)11.29
• Understanding Retrieval Robustness For Retrieval-augmented Image Captioning (2024)6.34
• Retrieval Augmentation For Deep Neural Networks (2021)5.84
• Improving Image Recognition By Retrieving From Web-scale Image-text Data (2023)9.41
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• Event-retriever: Event-aware Multimodal Image Retrieval For Realistic Captions (2025)0.00
• Dualcap: Enhancing Lightweight Image Captioning Via Dual Retrieval With Similar Scenes Visual Prompts (2025)0.00
• Retrieval-augmented Multimodal Language Modeling (2022)0.00