Towards Identity-aware Cross-modal Retrieval: A Dataset And A Baseline

Abstract

Recent advancements in deep learning have significantly enhanced content-based retrieval methods, notably through models like CLIP that map images and texts into a shared embedding space. However, these methods often struggle with domain-specific entities and long-tail concepts absent from their training data, particularly in identifying specific individuals. In this paper, we explore the task of identity-aware cross-modal retrieval, which aims to retrieve images of persons in specific contexts based on natural language queries. This task is critical in various scenarios, such as for searching and browsing personalized video collections or large audio-visual archives maintained by national broadcasters. We introduce a novel dataset, COCO Person FaceSwap (COCO-PFS), derived from the widely used COCO dataset and enriched with deepfake-generated faces from VGGFace2. This dataset addresses the lack of large-scale datasets needed for training and evaluating models for this task. Our experim

Stats

• citations0
• S2 citations—
• github stars5
• HF likes0
• heat score1.56
• arxiv keymessina2024towards

Related papers

• Learnable Pins: Cross-modal Embeddings For Person Identity (2018)15.22
• Automatic Synthetic Data And Fine-grained Adaptive Feature Alignment For Composed Person Retrieval (2023)3.75
• Voice-face Cross-modal Matching And Retrieval: A Benchmark (2019)0.00
• Improving Text-based Person Search Via Part-level Cross-modal Correspondence (2024)0.00
• Learnt Quasi-transitive Similarity For Retrieval From Large Collections Of Faces (2016)5.24
• Learning Context-aware Embedding For Person Search (2021)0.00
• CPCL: Cross-modal Prototypical Contrastive Learning For Weakly Supervised Text-based Person Retrieval (2024)0.00
• Deep Co-attention Based Comparators For Relative Representation Learning In Person Re-identification (2018)13.34