Neurocache: Efficient Vector Retrieval For Long-range Language Modeling

Abstract

This paper introduces Neurocache, an approach to extend the effective context size of large language models (LLMs) using an external vector cache to store its past states. Like recent vector retrieval approaches, Neurocache uses an efficient k-nearest-neighbor (kNN) algorithm to retrieve relevant past states and incorporate them into the attention process. Neurocache improves upon previous methods by (1) storing compressed states, which reduces cache size; (2) performing a single retrieval operation per token which increases inference speed; and (3) extending the retrieval window to neighboring states, which improves both language modeling and downstream task accuracy. Our experiments show the effectiveness of Neurocache both for models trained from scratch and for pre-trained models such as Llama2-7B and Mistral-7B when enhanced with the cache mechanism. We also compare Neurocache with text retrieval methods and show improvements in single-document question-answering and few-shot lear

Stats

• citations0
• S2 citations—
• github stars8
• HF likes0
• heat score1.91
• arxiv keysafaya2024neurocache

Related papers

• Retrievalattention: Accelerating Long-context LLM Inference Via Vector Retrieval (2024)0.00
• Pqcache: Product Quantization-based Kvcache For Long Context LLM Inference (2024)7.50
• Hashevict: A Pre-attention KV Cache Eviction Strategy Using Locality-sensitive Hashing (2024)0.00
• Lightretriever: A Llm-based Text Retrieval Architecture With Extremely Faster Query Inference (2025)0.00
• Why Do Nearest Neighbor Language Models Work? (2023)3.56
• Enhancing Question Answering Precision With Optimized Vector Retrieval And Instructions (2024)0.00
• Scaling Sparse And Dense Retrieval In Decoder-only Llms (2025)6.34
• Pylate: Flexible Training And Retrieval For Late Interaction Models (2025)3.58