FLAIR: VLM With Fine-grained Language-informed Image Representations

Abstract

CLIP has shown impressive results in aligning images and texts at scale. However, its ability to capture detailed visual features remains limited because CLIP matches images and texts at a global level. To address this issue, we propose FLAIR, Fine-grained Language-informed Image Representations, an approach that utilizes long and detailed image descriptions to learn localized image embeddings. By sampling diverse sub-captions that describe fine-grained details about an image, we train our vision-language model to produce not only global embeddings but also text-specific image representations. Our model introduces text-conditioned attention pooling on top of local image tokens to produce fine-grained image representations that excel at retrieving detailed image content. We achieve state-of-the-art performance on both, existing multimodal retrieval benchmarks, as well as, our newly introduced fine-grained retrieval task which evaluates vision-language models' ability to retrieve partial

Stats

• citations5
• S2 citations—
• github stars141
• HF likes0
• heat score10.14
• arxiv keyxiao2024flair

Related papers

• FG-CLIP: Fine-grained Visual And Textual Alignment (2025)5.75
• Finelip: Extending Clip's Reach Via Fine-grained Alignment With Longer Text Inputs (2025)6.34
• LLM2CLIP: Powerful Language Model Unlocks Richer Cross-modality Representation (2024)2.26
• Finetuning CLIP To Reason About Pairwise Differences (2024)0.00
• Linear Alignment Of Vision-language Models For Image Captioning (2023)0.00
• Detailclip: Injecting Image Details Into Clip's Feature Space (2022)0.00
• Finevit: Progressively Unlocking Fine-grained Perception With Dense Recaptions (2026)0.00
• Language-driven Fine-grained Retrieval (2025)0.00