Massively Multilingual Sentence Embeddings For Zero-shot Cross-lingual Transfer And Beyond

Abstract

We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and

Stats

• citations362
• S2 citations—
• github stars3661
• HF likes0
• heat score26.33
• arxiv keyartetxe2018massively

Related papers

• Towards Zero-shot Cross-lingual Image Retrieval And Tagging (2021)2.46
• Towards Zero-shot Cross-lingual Image Retrieval (2020)2.46
• On Cross-lingual Retrieval With Multilingual Text Encoders (2021)10.35
• Margin-based Parallel Corpus Mining With Multilingual Sentence Embeddings (2018)10.97
• Evaluating Multilingual Text Encoders For Unsupervised Cross-lingual Retrieval (2021)7.50
• Aligning Multilingual Word Embeddings For Cross-modal Retrieval Task (2019)2.26
• Modeling Sequential Sentence Relation To Improve Cross-lingual Dense Retrieval (2023)1.20
• LUSIFER: Language Universal Space Integration For Enhanced Multilingual Embeddings With Large Language Models (2025)0.00