Beyond Patch Aggregation: 3-pass Pyramid Indexing For Vision-enhanced Document Retrieval

Abstract

Document centric RAG pipelines usually begin with OCR, followed by brittle heuristics for chunking, table parsing, and layout reconstruction. These text first workflows are costly to maintain, sensitive to small layout shifts, and often lose the spatial cues that contain the answer. Vision first retrieval has emerged as a strong alternative. By operating directly on page images, systems like ColPali and ColQwen preserve structure and reduce pipeline complexity while achieving strong benchmark performance. However, these late interaction models tie retrieval to a specific vision backbone and require storing hundreds of patch embeddings per page, creating high memory overhead and complicating large scale deployment. We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic. VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction. Our three pass py

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyroy2025beyond

Related papers

• Spatially-grounded Document Retrieval Via Patch-to-region Relevance Propagation (2025)3.86
• Enhancing Document VQA Models Via Retrieval-augmented Generation (2025)0.00
• Colpali: Efficient Document Retrieval With Vision Language Models (2024)0.00
• Agenticocr: Parsing Only What You Need For Efficient Retrieval-augmented Generation (2026)0.00
• Vdocrag: Retrieval-augmented Generation Over Visually-rich Documents (2025)6.34
• Regionrag: Region-level Retrieval-augmented Generation For Visual Document Understanding (2025)0.00
• Modernvbert: Towards Smaller Visual Document Retrievers (2025)0.00
• Document Haystacks: Vision-language Reasoning Over Piles Of 1000+ Documents (2024)2.83