REVEAL: Retrieval-augmented Visual-language Pre-training With Multi-source Multimodal Knowledge Memory

Abstract

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. REVEAL consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that REVEAL achieves state-of-the-art results on visual ques

Stats

• citations65
• S2 citations—
• github stars0
• HF likes0
• heat score13.65
• arxiv keyhu2022reveal

Related papers

• End-to-end Knowledge Retrieval With Multi-modal Queries (2023)8.35
• Generative Cross-modal Retrieval: Memorizing Images In Multimodal Language Models For Retrieval And Beyond (2024)8.35
• Cross-modal Retrieval For Knowledge-based Visual Question Answering (2024)7.81
• Pre-training Multi-modal Dense Retrievers For Outside-knowledge Visual Question Answering (2023)7.50
• V-retrver: Evidence-driven Agentic Reasoning For Universal Multimodal Retrieval (2026)0.00
• Retrieval-augmented Multimodal Language Modeling (2022)0.00
• Pixel-grounded Retrieval For Knowledgeable Large Multimodal Models (2026)0.00
• RAVEN: Multitask Retrieval Augmented Vision-language Learning (2024)0.00