LEAF: Knowledge Distillation Of Text Embedding Models With Teacher-aligned Representations

Abstract

We present LEAF ("Lightweight Embedding Alignment Framework"), a knowledge distillation framework for text embedding models. A key distinguishing feature is that our distilled leaf models are aligned to their teacher. In the context of information retrieval, this allows for flexible asymmetric architectures where documents are encoded with the larger teacher model, while queries can be served with the smaller leaf models. We also show that leaf models automatically inherit MRL and robustness to output quantization whenever these properties are present in the teacher model, without explicitly training for them. To demonstrate the capability of our framework we publish leaf-ir, a 23M parameters information retrieval oriented text embedding model trained using LEAF, which sets a new state-of-the-art (SOTA) on BEIR, ranking \#1 on the public leaderboard for this benchmark and for models of its size. When run in asymmetric mode, its retrieval performance is further increased. Our scheme is

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvujanic2025leaf

Related papers

• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Knowledge Distillation In Document Retrieval (2019)0.00
• Learning Effective Representations For Retrieval Using Self-distillation With Adaptive Relevance Margins (2024)2.26
• Robustness In Both Domains: CLIP Needs A Robust Text Encoder (2025)2.16
• AMMKD: Adaptive Multimodal Multi-teacher Distillation For Lightweight Vision-language Models (2025)0.00
• Training Llms To Be Better Text Embedders Through Bidirectional Reconstruction (2025)0.00
• Data-efficient Ranking Distillation For Image Retrieval (2020)0.00
• Translate-distill: Learning Cross-language Dense Retrieval By Translation And Distillation (2024)8.60