Livevectorlake: A Real-time Versioned Knowledge Base Architecture For Streaming Vector Updates And Temporal Retrieval
2025 · Tarun Prajapati
Abstract
Modern Retrieval-Augmented Generation (RAG) systems struggle with a fundamental architectural tension: vector indices are optimized for query latency but poorly handle continuous knowledge updates, while data lakes excel at versioning but introduce query latency penalties. We introduce LiveVectorLake, a dual-tier temporal knowledge base architecture that enables real-time semantic search on current knowledge while maintaining complete version history for compliance, auditability, and point-in-time retrieval. The system introduces three core architectural contributions: (1) Content-addressable chunk-level synchronization using SHA-256 hashing for deterministic change detection without external state tracking; (2) Dual-tier storage separating hot-tier vector indices (Milvus with HNSW) from cold-tier columnar versioning (Delta Lake with Parquet), optimizing query latency and storage cost independently; (3) Temporal query routing enabling point-in-time knowledge retrieval via delta-version
Authors
(none)
Tags
Stats
Related papers
- Self-aware Vector Embeddings For Retrieval-augmented Generation: A Neuroscience-inspired Framework For Temporal, Confidence-weighted, And Relational Knowledge (2026)0.00
- Cracking Vector Search Indexes (2025)3.58
- Real-time Indexing For Large-scale Recommendation By Streaming Vector Quantization Retriever (2025)2.26
- Spatcode: Rotary-based Unified Encoding Framework For Efficient Spatiotemporal Vector Retrieval (2026)0.00
- Zoom: Ssd-based Vector Search For Optimizing Accuracy, Latency And Memory (2018)0.00
- MQRLD: A Multimodal Data Retrieval Platform With Query-aware Feature Representation And Learned Index Based On Data Lake (2024)8.35
- LUMA-RAG: Lifelong Multimodal Agents With Provably Stable Streaming Alignment (2025)0.00
- Vectorsearch: Enhancing Document Retrieval With Semantic Embeddings And Optimized Search (2024)0.00