Interpreting Dense Retrieval As Mixture Of Topics

Abstract

Dense Retrieval (DR) reaches state-of-the-art results in first-stage retrieval, but little is known about the mechanisms that contribute to its success. Therefore, in this work, we conduct an interpretation study of recently proposed DR models. Specifically, we first discretize the embeddings output by the document and query encoders. Based on the discrete representations, we analyze the attribution of input tokens. Both qualitative and quantitative experiments are carried out on public test collections. Results suggest that DR models pay attention to different aspects of input and extract various high-level topic representations. Therefore, we can regard the representations learned by DR models as a mixture of high-level topics.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhan2021interpreting

Related papers

• Few-shot Conversational Dense Retrieval (2021)16.68
• Disentangled Modeling Of Domain And Relevance For Adaptable Dense Retrieval (2022)0.00
• Mixture Of Experts Approaches In Dense Retrieval Tasks (2025)0.95
• What Are You Token About? Dense Retrieval As Distributions Over The Vocabulary (2022)8.09
• Efficiently Teaching An Effective Dense Retriever With Balanced Topic Aware Sampling (2021)17.07
• How To Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (2023)11.39
• Dense Passage Retrieval: Is It Retrieving? (2024)6.34
• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00