Prototype-based Aleatoric Uncertainty Quantification For Cross-modal Retrieval

Abstract

Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchm

Stats

• citations2
• S2 citations—
• github stars28
• HF likes0
• heat score6.50
• arxiv keyli2023prototype

Related papers

• Uncertainty-based Cross-modal Retrieval With Probabilistic Representations (2022)0.00
• Exploring Uncertainty In Conditional Multi-modal Retrieval Systems (2019)0.00
• Probabilistic Embeddings For Cross-modal Retrieval (2021)21.70
• Prototype-based Semantic Consistency Alignment For Domain Adaptive Retrieval (2025)0.00
• Prototypes Are Balanced Units For Efficient And Effective Partially Relevant Video Retrieval (2025)0.00
• Reliability-aware Prediction Via Uncertainty Learning For Person Image Retrieval (2022)8.35
• Prototype-enhanced Confidence Modeling For Cross-modal Medical Image-report Retrieval (2025)0.00
• Learning To Rematch Mismatched Pairs For Robust Cross-modal Retrieval (2024)13.82