Large-scale Adversarial Training For Vision-and-language Representation Learning

Abstract

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygan2020large

Related papers

• Villa: Fine-grained Vision-language Representation Learning From Real-world Data (2023)8.82
• Vilbert: Pretraining Task-agnostic Visiolinguistic Representations For Vision-and-language Tasks (2019)0.00
• CAVL: Learning Contrastive And Adaptive Representations Of Vision And Language (2023)0.00
• Unsupervised Vision-and-language Pre-training Via Retrieval-based Multi-granular Alignment (2022)10.48
• Infusing Fine-grained Visual Knowledge To Vision-language Models (2025)0.00
• 12-in-1: Multi-task Vision And Language Representation Learning (2019)17.85
• VLMAE: Vision-language Masked Autoencoder (2022)0.00
• Personalization Toolkit: Training Free Personalization Of Large Vision Language Models (2026)0.00