Retrieval-augmented Image Captioning

Abstract

Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore, as opposed to the image alone. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions. Experimental results on the COCO dataset show that image captioning can be effectively formulated from this new perspective. Our model, named EXTRA, benefits from using captions retrieved from the training dataset, and it can also benefit from using an external dataset without the need for retraining. Ablation studies show that retrieving a sufficient number of captions (e.g., k=5) can improve captioning quality. Our work contributes towards using pretrained V&L encoders f

Stats

• citations31
• S2 citations—
• github stars0
• HF likes0
• heat score11.29
• arxiv keyramos2023retrieval

Related papers

• Towards Retrieval-augmented Architectures For Image Captioning (2024)9.41
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• CLIPS: An Enhanced CLIP Framework For Learning With Synthetic Captions (2024)0.00
• Deep Image Representations Using Caption Generators (2017)0.00
• Dualcap: Enhancing Lightweight Image Captioning Via Dual Retrieval With Similar Scenes Visual Prompts (2025)0.00
• Understanding Retrieval Robustness For Retrieval-augmented Image Captioning (2024)6.34
• Mllms-augmented Visual-language Representation Learning (2023)0.00
• Dreamlip: Language-image Pre-training With Long Captions (2024)10.61