Bootstrapped Pre-training With Dynamic Identifier Prediction For Generative Retrieval

Abstract

Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. However, the full power of pre-training for generative retrieval remains underexploited due to its reliance on pre-defined static document identifiers, which may not align with evolving model parameters. In this work, we introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing memorization of the corpus. BootRet involves three key training phases: (i) initial identifier generation, (ii) pre-training via corpus indexing and relevance prediction tasks, and (iii) bootstrapping for identifier updates. To facilitate the pre-training phase, we further introdu

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keytang2024bootstrapped

Related papers

• Learning To Tokenize For Generative Retrieval (2023)4.52
• Continual Learning For Generative Retrieval Over Dynamic Corpora (2023)11.49
• Generative Retrieval Meets Multi-graded Relevance (2024)2.26
• GLEN: Generative Retrieval Via Lexical Index Learning (2023)9.29
• Generative Retrieval As Multi-vector Dense Retrieval (2024)8.60
• Lightweight And Direct Document Relevance Optimization For Generative Information Retrieval (2025)4.52
• Scalable And Effective Generative Information Retrieval (2023)10.48
• Planning Ahead In Generative Retrieval: Guiding Autoregressive Generation Through Simultaneous Decoding (2024)8.82