Diffusion-pretrained Dense And Contextual Embeddings

Abstract

In this report, we introduce pplx-embed, a family of multilingual embedding models that employ multi-stage contrastive learning on a diffusion-pretrained language model backbone for web-scale retrieval. By leveraging bidirectional attention through diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard retrieval, and pplx-embed-context-v1 for contextualized embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB benchmark. Beyond public benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeslami2026diffusion

Related papers

• Unsupervised Context Aware Sentence Representation Pretraining For Multi-lingual Dense Retrieval (2022)3.58
• Context-adaptive Multi-prompt Embedding With Large Language Models For Vision-language Alignment (2025)0.00
• Dewey Long Context Embedding Model: A Technical Report (2025)0.00
• Context Is Gold To Find The Gold Passage: Evaluating And Training Contextual Document Embeddings (2025)5.62
• CEQE: Contextualized Embeddings For Query Expansion (2021)10.35
• M3-embedding: Multi-linguality, Multi-functionality, Multi-granularity Text Embeddings Through Self-knowledge Distillation (2024)19.54
• Rzenembed: Towards Comprehensive Multimodal Retrieval (2025)0.00
• Investigating Multi-layer Representations For Dense Passage Retrieval (2025)0.00