Matryoshka-adaptor: Unsupervised And Supervised Tuning For Smaller Embedding Dimensions

Abstract

Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval. While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost. To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings. Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness. Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs. Also, it exhibits efficacy in both unsupervised and supervised

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyyoon2024matryoshka

Related papers

• SMEC: Rethinking Matryoshka Representation Learning For Retrieval Embedding Compression (2025)0.00
• Search-adaptor: Embedding Customization For Information Retrieval (2023)0.00
• Multiway-adapater: Adapting Large-scale Multi-modal Models For Scalable Image-text Retrieval (2023)0.00
• Federated Learning With Ad-hoc Adapter Insertions: The Case Of Soft-embeddings For Training Classifier-as-retriever (2025)0.00
• Efficient Temporal-aware Matryoshka Adaptation For Temporal Information Retrieval (2026)0.00
• Compressed Concatenation Of Small Embedding Models (2025)0.00
• Matryoshka Representation Learning (2022)12.37
• 2D Matryoshka Training For Information Retrieval (2024)4.06