Cracking Vector Search Indexes

Abstract

Retrieval Augmented Generation (RAG) uses vector databases to expand the expertise of an LLM model without having to retrain it. The idea can be applied over data lakes, leading to the notion of embedding data lakes, i.e., a pool of vector databases ready to be used by RAGs. The key component in these systems is the indexes enabling Approximated Nearest Neighbor Search (ANNS). However, in data lakes, one cannot realistically expect to build indexes for every dataset. Thus, we propose an adaptive, partition-based index, CrackIVF, that performs much better than up-front index building. CrackIVF starts answering as a small index, and only expands to improve performance as it sees enough queries. It does so by progressively adapting the index to the query workload. That way, queries can be answered right away without having to build a full index first. After seeing enough queries, CrackIVF will produce an index comparable to those built with conventional techniques. CrackIVF can often answ

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keymageirakos2025cracking

Related papers

• SRAG: RAG With Structured Data Improves Vector Retrieval (2026)0.00
• Ragdb: A Zero-dependency, Embeddable Architecture For Multimodal Retrieval-augmented Generation On The Edge (2025)0.00
• FRAG: Toward Federated Vector Database Management For Collaborative And Secure Retrieval-augmented Generation (2024)0.00
• Exqutor: Extended Query Optimizer For Vector-augmented Analytical Queries (2025)0.00
• Livevectorlake: A Real-time Versioned Knowledge Base Architecture For Streaming Vector Updates And Temporal Retrieval (2025)0.95
• 4bit-quantization In Vector-embedding For RAG (2025)6.21
• Edgerag: Online-indexed RAG For Edge Devices (2024)0.00
• Self-aware Vector Embeddings For Retrieval-augmented Generation: A Neuroscience-inspired Framework For Temporal, Confidence-weighted, And Relational Knowledge (2026)0.00