Lightningdot: Pre-training Visual-semantic Embeddings For Real-time Image-text Retrieval

Abstract

Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computation cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by pre-training on three novel learning objectives, extracting feature indexes offline, and employing instant dot-product matching with further re-ranking, which significantl

Stats

• citations66
• S2 citations—
• github stars72
• HF likes0
• heat score17.42
• arxiv keysun2021lightningdot

Related papers

• Lexlip: Lexicon-bottlenecked Language-image Pre-training For Large-scale Image-text Retrieval (2023)10.85
• Dynamic Contrastive Distillation For Image-text Retrieval (2022)11.76
• Hivlp: Hierarchical Vision-language Pre-training For Fast Image-text Retrieval (2022)0.00
• Vldeformer: Vision-language Decomposed Transformer For Fast Cross-modal Retrieval (2021)10.21
• COTS: Collaborative Two-stream Vision-language Pre-training Model For Cross-modal Retrieval (2022)13.60
• Imagebert: Cross-modal Pre-training With Large-scale Weak-supervised Image-text Data (2020)0.00
• Litevl: Efficient Video-language Learning With Enhanced Spatial-temporal Modeling (2022)6.34
• Thinking Fast And Slow: Efficient Text-to-visual Retrieval With Transformers (2021)15.16