Twinbert: Distilling Knowledge To Twin-structured BERT Models For Efficient Retrieval

Abstract

Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources, which hinders the application in low-latency IR systems. We present TwinBERT model for effective and efficient retrieval, which has twin-structured BERT-like encoders to represent query and document respectively and a crossing layer to combine the embeddings and produce a similarity score. Different from BERT, where the two input sentences are concatenated and encoded together, TwinBERT decouples them during encoding and produces the embeddings for query and document independently, which allows document embeddings to be pre-computed offline and cached in memory. Thereupon, the computation left for run-time is from the query encoding and query-document crossing only. This single change can save large amount of computation time and resources, and therefore significantly improve serving efficiency. Moreover, a f

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylu2020twinbert

Related papers

• Diagnosing BERT With Retrieval Heuristics (2022)10.21
• Colbert: Efficient And Effective Passage Search Via Contextualized Late Interaction Over BERT (2020)0.00
• Predicting Efficiency/effectiveness Trade-offs For Dense Vs. Sparse Retrieval Strategy Selection (2021)11.29
• Transfer Learning Approaches For Building Cross-language Dense Retrieval Models (2022)10.97
• Colbertv2: Effective And Efficient Retrieval Via Lightweight Late Interaction (2021)17.46
• Pre-training Tasks For Embedding-based Large-scale Retrieval (2020)0.00
• Query Encoder Distillation Via Embedding Alignment Is A Strong Baseline Method To Boost Dense Retriever Online Efficiency (2023)0.00
• SDR: Efficient Neural Re-ranking Using Succinct Document Representation (2021)3.58