FOR: Finetuning For Object Level Open Vocabulary Image Retrieval

Abstract

As working with large datasets becomes standard, the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain, balancing accuracy and efficiency through additional post-processing. In this work, we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval, which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task, and its coupling within a multi-objective training framework. Together, these design choices result in a significant increase in accuracy, showcasing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally, we demonstrate that FOR is also effective in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylevi2024for

Related papers

• Object-centric Open-vocabulary Image-retrieval With Aggregated Features (2023)0.00
• Describe, Adapt And Combine: Empowering CLIP Encoders For Open-set 3D Object Retrieval (2025)2.51
• Revisiting Human-in-the-loop Object Retrieval With Pre-trained Vision Transformers (2026)0.00
• FORB: A Flat Object Retrieval Benchmark For Universal Image Embedding (2023)2.23
• Infusing Fine-grained Visual Knowledge To Vision-language Models (2025)0.00
• Find Your Needle: Small Object Image Retrieval Via Multi-object Attention Optimization (2025)0.00
• Object-aware Video-language Pre-training For Retrieval (2021)16.14
• EFSA: Episodic Few-shot Adaptation For Text-to-image Retrieval (2024)0.00