Comparative Analysis Of Lion And Adamw Optimizers For Cross-encoder Reranking With Minilm, GTE, And Modernbert

Abstract

Modern information retrieval systems often employ a two-stage pipeline: an efficient initial retrieval stage followed by a computationally intensive reranking stage. Cross-encoders have shown strong effectiveness for reranking due to their deep analysis of query-document pairs. This paper studies the impact of the Lion optimizer, a recent alternative to AdamW, during fine-tuning of cross-encoder rerankers. We fine-tune three transformer models-MiniLM, GTE, and ModernBERT-on the MS MARCO passage ranking dataset using both optimizers. GTE and ModernBERT support extended context lengths (up to 8192 tokens). We evaluate effectiveness using TREC 2019 Deep Learning Track and MS MARCO dev set (MRR@10). Experiments, run on the Modal cloud platform, reveal that ModernBERT with Lion achieves the best NDCG@10 (0.7225) and MAP (0.5121) on TREC DL 2019, while MiniLM with Lion ties ModernBERT for MRR@10 (0.5988) on MS MARCO dev. Lion also provides superior GPU efficiency, improving utilization by 2.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykumar2025comparative

Related papers

• How Different Are Pre-trained Transformers For Text Ranking? (2022)7.81
• Shallow Cross-encoders For Low-latency Retrieval (2024)2.26
• CODER: An Efficient Framework For Improving Retrieval Through Contextual Document Embedding Reranking (2021)7.16
• MICE: Minimal Interaction Cross-encoders For Efficient Re-ranking (2026)0.00
• Drowning In Documents: Consequences Of Scaling Reranker Inference (2024)0.00
• Supervised Fine-tuning Or Contrastive Learning? Towards Better Multimodal LLM Reranking (2025)0.00
• Rethinking Hybrid Retrieval: When Small Embeddings And LLM Re-ranking Beat Bigger Models (2025)0.00
• Comparing Neighbors Together Makes It Easy: Jointly Comparing Multiple Candidates For Efficient And Effective Retrieval (2024)4.52