Multimodal Prototypical Networks For Few-shot Learning

Abstract

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxf

Stats

• citations91
• S2 citations—
• github stars0
• HF likes0
• heat score14.73
• arxiv keypahde2020multimodal

Related papers

• Text-to-image Generation Via Implicit Visual Guidance And Hypernetwork (2022)0.00
• FLEX-CLIP: Feature-level Generation Network Enhanced CLIP For X-shot Cross-modal Retrieval (2024)0.00
• Look, Imagine And Match: Improving Textual-visual Cross-modal Retrieval With Generative Models (2017)18.52
• Multimodal Contrastive Training For Visual Representation Learning (2021)16.32
• ASIF: Coupled Data Turns Unimodal Models To Multimodal Without Training (2022)0.00
• Single-branch Network For Multimodal Training (2023)13.26
• Revisiting Metric Learning For Few-shot Image Classification (2019)14.90
• Multimodal Clustering Networks For Self-supervised Learning From Unlabeled Videos (2021)13.28