TULIP: Token-length Upgraded CLIP

Abstract

We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such

Stats

• citations0
• S2 citations—
• github stars32
• HF likes0
• heat score3.04
• arxiv keynajdenkoska2024tulip

Related papers

• Finelip: Extending Clip's Reach Via Fine-grained Alignment With Longer Text Inputs (2025)6.34
• Long-clip: Unlocking The Long-text Capability Of CLIP (2024)14.90
• LLM2CLIP: Powerful Language Model Unlocks Richer Cross-modality Representation (2024)2.26
• Lotlip: Improving Language-image Pre-training For Long Text Understanding (2024)2.26
• Videoclip-xl: Advancing Long Description Understanding For Video CLIP Models (2024)8.35
• Linear Alignment Of Vision-language Models For Image Captioning (2023)0.00
• GOAL: Global-local Object Alignment Learning (2025)2.26
• CLIPS: An Enhanced CLIP Framework For Learning With Synthetic Captions (2024)0.00