COCO-DR: Combating Distribution Shifts In Zero-shot Dense Retrieval With Contrastive And Distributionally Robust Learning
2022 · Yue Yu, Chenyan Xiong, Si Sun, et al.
Abstract
We present a new zero-shot dense retrieval (ZeroDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios. To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning. To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning. COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark. At BERT Base scale, COCO-DR Base outperforms other ZeroDR models with 60x larger size. At BERT Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500x more parameters. Our analysis show the correlation between COCO-DR's effectiveness
Authors
(none)
Tags
Stats
Related papers
- How To Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (2023)11.39
- Few-shot Conversational Dense Retrieval (2021)16.68
- Zero-shot Dense Retrieval With Momentum Adversarial Domain Invariant Representations (2021)6.77
- Laprador: Unsupervised Pretrained Dense Retriever For Zero-shot Text Retrieval (2022)8.82
- A Representation Sharpening Framework For Zero Shot Dense Retrieval (2025)0.00
- A Distributed Collaborative Retrieval Framework Excelling In All Queries And Corpora Based On Zero-shot Rank-oriented Automatic Evaluation (2024)0.00
- Disentangled Modeling Of Domain And Relevance For Adaptable Dense Retrieval (2022)0.00
- Boot And Switch: Alternating Distillation For Zero-shot Dense Retrieval (2023)0.00