Understanding Retrieval Robustness For Retrieval-augmented Image Captioning

Abstract

Recent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient, lightweight models with strong domain-transfer capabilities. While these models demonstrate the success of retrieval augmentation, retrieval models are still far from perfect in practice: the retrieved information can sometimes mislead the model, resulting in incorrect generation and worse performance. In this paper, we analyze the robustness of a retrieval-augmented captioning model SmallCap. Our analysis shows that the model is sensitive to tokens that appear in the majority of the retrieved captions, and the input attribution shows that those tokens are likely copied into the generated output. Given these findings, we propose to train the model by sampling retrieved captions from more diverse sets. This decreases the chance that the model learns to copy majority tokens, and improves both in-domain and cross-domain performance.

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyli2024understanding

Related papers

• Towards Retrieval-augmented Architectures For Image Captioning (2024)9.41
• Dualcap: Enhancing Lightweight Image Captioning Via Dual Retrieval With Similar Scenes Visual Prompts (2025)0.00
• Retrieval-augmented Image Captioning (2023)11.29
• Benchmark Granularity And Model Robustness For Image-text Retrieval (2024)0.00
• Captions Are Worth A Thousand Words: Enhancing Product Retrieval With Pretrained Image-to-text Models (2024)0.00
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• Noise-robust Dense Retrieval Via Contrastive Alignment Post Training (2023)0.00
• Event-retriever: Event-aware Multimodal Image Retrieval For Realistic Captions (2025)0.00