OLIVE: Object Level In-context Visual Embeddings

Abstract

Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with fine-grained object-level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object-level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitati

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyossowski2024olive

Related papers

• Objembed: Towards Universal Multimodal Object Embeddings (2026)0.00
• Give: Guiding Visual Encoder To Perceive Overlooked Information (2024)0.00
• Prompting Large Vision-language Models For Compositional Reasoning (2024)0.00
• VIRTUE: Visual-interactive Text-image Universal Embedder (2025)0.00
• Infusing Fine-grained Visual Knowledge To Vision-language Models (2025)0.00
• Leveraging Retrieval-augmented Tags For Large Vision-language Understanding In Complex Scenes (2024)0.00
• Reminding Multimodal Large Language Models Of Object-aware Knowledge With Retrieved Tags (2024)0.00
• Vlm2geovec: Toward Universal Multimodal Embeddings For Remote Sensing (2025)0.00