Pooling And Attention: What Are Effective Designs For Llm-based Embedding Models?

Abstract

The significant advancements of Large Language Models (LLMs) in generative tasks have led to a growing body of work exploring LLM-based embedding models. While these models, employing different pooling and attention strategies, have achieved state-of-the-art performance on public embedding benchmarks, questions still arise about what constitutes an effective design for LLM-based embedding models. However, these models are often trained on different datasets, using different LLM base models or training settings. Moreover, evaluations on public embedding benchmarks often fail to report statistical significance, making it difficult to determine which designs truly contribute to final performance. This complicates the process for practitioners seeking optimal training recipes for LLM-based embedding models. In this study, we conduct a large-scale experiment by training a series of LLM-based embedding models using the same training data and base model but differing in their pooling and atte

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2024pooling

Related papers

• Nv-embed: Improved Techniques For Training Llms As Generalist Embedding Models (2024)0.00
• Making Large Language Models Efficient Dense Retrievers (2025)0.00
• Scaling Sparse And Dense Retrieval In Decoder-only Llms (2025)6.34
• Training Llms To Be Better Text Embedders Through Bidirectional Reconstruction (2025)0.00
• Rethinking Hybrid Retrieval: When Small Embeddings And LLM Re-ranking Beat Bigger Models (2025)0.00
• Llave: Large Language And Vision Embedding Models With Hardness-weighted Contrastive Learning (2025)3.58
• Llm-augmented Retrieval: Enhancing Retrieval Models Through Language Models And Doc-level Embedding (2024)0.00
• U-MARVEL: Unveiling Key Factors For Universal Multimodal Retrieval Via Embedding Learning With Mllms (2025)3.11