Generalization Properties Of Retrieval-based Models

Abstract

Many modern high-performing machine learning models such as GPT-3 primarily rely on scaling up models, e.g., transformer networks. Simultaneously, a parallel line of work aims to improve the model performance by augmenting an input instance with other (labeled) instances during inference. Examples of such augmentations include task-specific prompts and similar examples retrieved from the training data by a nonparametric component. Remarkably, retrieval-based methods have enjoyed success on a wide range of problems, ranging from standard natural language processing and vision tasks to protein folding, as demonstrated by many recent efforts, including WebGPT and AlphaFold. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. In this paper, we present a formal treatment of retrieval-based models to characterize their generalization ability. In particular, we focus on two classes of retrieval-based classifica

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybasu2022generalization

Related papers

• Modular Retrieval For Generalization And Interpretation (2023)0.00
• Large Dual Encoders Are Generalizable Retrievers (2021)14.69
• Scaling Laws For Dense Retrieval (2024)10.07
• Generative Retrieval As Dense Retrieval (2023)0.00
• How To Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (2023)11.39
• Evaluating The Effectiveness And Scalability Of Llm-based Data Augmentation For Retrieval (2025)0.00
• Does Generative Retrieval Overcome The Limitations Of Dense Retrieval? (2025)0.00
• Retrieval-enhanced Machine Learning (2022)11.93