SMEC: Rethinking Matryoshka Representation Learning For Retrieval Embedding Compression

Abstract

Large language models (LLMs) generate high-dimensional embeddings that capture rich semantic and syntactic information. However, high-dimensional embeddings exacerbate computational complexity and storage requirements, thereby hindering practical deployment. To address these challenges, we propose a novel training framework named Sequential Matryoshka Embedding Compression (SMEC). This framework introduces the Sequential Matryoshka Representation Learning(SMRL) method to mitigate gradient variance during training, the Adaptive Dimension Selection (ADS) module to reduce information degradation during dimension pruning, and the Selectable Cross-batch Memory (S-XBM) module to enhance unsupervised learning between high- and low-dimensional embeddings. Experiments on image, text, and multimodal datasets demonstrate that SMEC achieves significant dimensionality reduction while maintaining performance. For instance, on the BEIR dataset, our approach improves the performance of compressed LLM2

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2025smec

Related papers

• CREM: Compression-driven Representation Enhancement For Multimodal Retrieval And Comprehension (2026)0.00
• Matryoshka-adaptor: Unsupervised And Supervised Tuning For Smaller Embedding Dimensions (2024)2.26
• Compressed Concatenation Of Small Embedding Models (2025)0.00
• Beyond Matryoshka: Revisiting Sparse Coding For Adaptive Representation (2025)4.30
• Magic-mm-embedding: Towards Visual-token-efficient Universal Multimodal Embedding With Mllms (2026)0.00
• Matryoshka Representation Learning (2022)12.37
• Compressing Then Matching: An Efficient Pre-training Paradigm For Multimodal Embedding (2025)0.00
• Rethinking Hybrid Retrieval: When Small Embeddings And LLM Re-ranking Beat Bigger Models (2025)0.00