Investigating Multi-layer Representations For Dense Passage Retrieval

Abstract

Dense retrieval models usually adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model usually contain different kinds of linguistic knowledge, and behave differently during fine-tuning. Therefore, we propose to investigate utilizing representations from multiple encoder layers to make up the representation of a document, which we denote Multi-layer Representations (MLR). We first investigate how representations in different layers affect MLR's performance under the multi-vector retrieval setting, and then propose to leverage pooling strategies to reduce multi-vector models to single-vector ones to improve retrieval efficiency. Experiments demonstrate the effectiveness of MLR over dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting, as well as demonstrate that it works well with other advanced training techniques such as retrie

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxie2025investigating

Related papers

• Pseudo-relevance Feedback For Multiple Representation Dense Retrieval (2021)12.93
• Multi-view Document Representation Learning For Open-domain Dense Retrieval (2022)10.21
• Llm-augmented Retrieval: Enhancing Retrieval Models Through Language Models And Doc-level Embedding (2024)0.00
• Reinpool: Reinforcement Learning Pooling Multi-vector Embeddings For Retrieval System (2026)0.00
• Learning Diverse Document Representations With Deep Query Interactions For Dense Retrieval (2022)2.51
• Unifier: A Unified Retriever For Large-scale Retrieval (2022)7.50
• Transfer Learning Approaches For Building Cross-language Dense Retrieval Models (2022)10.97
• On Approximate Nearest Neighbour Selection For Multi-stage Dense Retrieval (2021)8.35