You Can't Pick Your Neighbors, Or Can You? When And How To Rely On Retrieval In The \(k\)nn-lm

Abstract

Retrieval-enhanced language models (LMs), which condition their predictions on text retrieved from large external datastores, have recently shown significant perplexity improvements compared to standard LMs. One such approach, the \(k\)NN-LM, interpolates any existing LM's predictions with the output of a \(k\)-nearest neighbors model and requires no additional training. In this paper, we explore the importance of lexical and semantic matching in the context of items retrieved by \(k\)NN-LM. We find two trends: (1) the presence of large overlapping \(n\)-grams between the datastore and evaluation set plays an important factor in strong performance, even when the datastore is derived from the training data; and (2) the \(k\)NN-LM is most beneficial when retrieved items have high semantic similarity with the query. Based on our analysis, we define a new formulation of the \(k\)NN-LM that uses retrieval quality to assign the interpolation coefficient. We empirically measure the effectiven

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keydrozdov2022you

Related papers

• Why Do Nearest Neighbor Language Models Work? (2023)3.56
• Neurocache: Efficient Vector Retrieval For Long-range Language Modeling (2024)1.91
• NUDGE: Lightweight Non-parametric Fine-tuning Of Embeddings For Retrieval (2024)0.00
• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00
• Scalingnote: Scaling Up Retrievers With Large Language Models For Real-world Dense Retrieval (2024)0.00
• Scaling Sparse And Dense Retrieval In Decoder-only Llms (2025)6.34
• Enhancing The Ranking Context Of Dense Retrieval Methods Through Reciprocal Nearest Neighbors (2023)4.52
• Feasibility Based Large Margin Nearest Neighbor Metric Learning (2016)0.00