Optimization Of Latent-space Compression Using Game-theoretic Techniques For Transformer-based Vector Search

Abstract

Vector similarity search plays a pivotal role in modern information retrieval systems, especially when powered by transformer-based embeddings. However, the scalability and efficiency of such systems are often hindered by the high dimensionality of latent representations. In this paper, we propose a novel game-theoretic framework for optimizing latent-space compression to enhance both the efficiency and semantic utility of vector search. By modeling the compression strategy as a zero-sum game between retrieval accuracy and storage efficiency, we derive a latent transformation that preserves semantic similarity while reducing redundancy. We benchmark our method against FAISS, a widely-used vector search library, and demonstrate that our approach achieves a significantly higher average similarity (0.9981 vs. 0.5517) and utility (0.8873 vs. 0.5194), albeit with a modest increase in query time. This trade-off highlights the practical value of game-theoretic latent compression in high-utili

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyagrawal2025optimization

Related papers

• Vectorsearch: Enhancing Document Retrieval With Semantic Embeddings And Optimized Search (2024)0.00
• Leanvec: Searching Vectors Faster By Making Them Fit (2023)0.00
• Semantic Vector Encoding And Similarity Search Using Fulltext Search Engines (2017)6.77
• Lossless Compression Of Vector Ids For Approximate Nearest Neighbor Search (2025)11.11
• Thinking Fast And Slow: Efficient Text-to-visual Retrieval With Transformers (2021)15.16
• Connecting Compression Spaces With Transformer For Approximate Nearest Neighbor Search (2021)4.52
• Experimental Analysis Of Large-scale Learnable Vector Storage Compression (2023)7.50
• Zoom: Ssd-based Vector Search For Optimizing Accuracy, Latency And Memory (2018)0.00