Spatially-grounded Document Retrieval Via Patch-to-region Relevance Propagation

Abstract

Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B

Stats

• citations0
• S2 citations—
• github stars84
• HF likes0
• heat score3.86
• arxiv keygeorgiou2025spatially

Related papers

• Beyond Patch Aggregation: 3-pass Pyramid Indexing For Vision-enhanced Document Retrieval (2025)0.00
• Regionrag: Region-level Retrieval-augmented Generation For Visual Document Understanding (2025)0.00
• Attention Grounded Enhancement For Visual Document Retrieval (2025)0.00
• Colpali: Efficient Document Retrieval With Vision Language Models (2024)0.00
• Enhancing Document VQA Models Via Retrieval-augmented Generation (2025)0.00
• Patch-wise Retrieval: A Bag Of Practical Techniques For Instance-level Matching (2025)0.00
• Visual Late Chunking: An Empirical Study Of Contextual Chunking For Efficient Visual Document Retrieval (2026)0.00
• Pixel-grounded Retrieval For Knowledgeable Large Multimodal Models (2026)0.00