Continual Learning In Cross-modal Retrieval

Abstract

Multimodal representations and continual learning are two areas closely related to human intelligence. The former considers the learning of shared representation spaces where information from different modalities can be compared and integrated (we focus on cross-modal retrieval between language and visual representations). The latter studies how to prevent forgetting a previously learned task when learning a new one. While humans excel in these two aspects, deep neural networks are still quite limited. In this paper, we propose a combination of both problems into a continual cross-modal retrieval setting, where we study how the catastrophic interference caused by new tasks impacts the embedding spaces and their cross-modal alignment required for effective retrieval. We propose a general framework that decouples the training, indexing and querying stages. We also identify and study different factors that may lead to forgetting, and propose tools to alleviate it. We found that the indexi

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keywang2021continual

Related papers

• Generative Cross-modal Retrieval: Memorizing Images In Multimodal Language Models For Retrieval And Beyond (2024)8.35
• Cross-modal Retrieval: A Systematic Review Of Methods And Future Directions (2023)12.81
• Deep Reversible Consistency Learning For Cross-modal Retrieval (2025)7.81
• Generalized Contrastive Learning For Universal Multimodal Retrieval (2025)0.00
• Advancing Continual Lifelong Learning In Neural Information Retrieval: Definition, Dataset, Framework, And Empirical Evaluation (2023)6.77
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• Learning Joint Embedding For Cross-modal Retrieval (2019)5.84