Realsyn: An Effective And Scalable Multimodal Interleaved Document Transformation Paradigm

Abstract

After pre-training on extensive image-text pairs, Contrastive Language-Image Pre-training (CLIP) demonstrates promising performance on a wide variety of benchmarks. However, a substantial volume of multimodal interleaved documents remains underutilized for contrastive vision-language representation learning. To fully leverage these unpaired documents, we initially establish a Real-World Data Extraction pipeline to extract high-quality images and texts. Then we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant realistic texts. To further enhance fine-grained visual information, we propose an image semantic augmented generation module for synthetic text production. Furthermore, we employ a semantic balance sampling strategy to improve dataset diversity, enabling better learning of long-tail concepts. Based on these innovations, we construct RealSyn, a dataset combining realistic and synthetic texts, available in three scales: 1

Stats

• citations0
• S2 citations—
• github stars39
• HF likes0
• heat score3.20
• arxiv keygu2025realsyn

Related papers

• CLIPS: An Enhanced CLIP Framework For Learning With Synthetic Captions (2024)0.00
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00
• Robust Cross-modal Representation Learning With Progressive Self-distillation (2022)12.33
• Exploring A Unified Vision-centric Contrastive Alternatives On Multi-modal Web Documents (2025)1.69
• Automatic Synthesis Of High-quality Triplet Data For Composed Image Retrieval (2025)0.00
• Tripletclip: Improving Compositional Reasoning Of CLIP Via Synthetic Vision-language Negatives (2024)4.52
• Linking Representations With Multimodal Contrastive Learning (2023)0.00
• Learning The Visualness Of Text Using Large Vision-language Models (2023)4.52