Learning Retrieval Models With Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyformal2026learning

Related papers

• Mistral-splade: Llms For Better Learned Sparse Retrieval (2024)0.00
• CSPLADE: Learned Sparse Retrieval With Causal Language Models (2025)0.00
• Interpret And Control Dense Retrieval With Sparse Latent Features (2024)2.26
• The Role Of Vocabularies In Learning Sparse Representations For Ranking (2025)0.00
• Decoding Dense Embeddings: Sparse Autoencoders For Interpreting And Discretizing Dense Retrieval (2025)0.00
• On The Challenges And Opportunities Of Learned Sparse Retrieval For Code (2026)0.00
• LANGSAE EDITING: Improving Multilingual Information Retrieval Via Post-hoc Language Identity Removal (2026)0.00
• Sparse Autoencoders For Interpretable Medical Image Representation Learning (2026)0.78